[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:57:10 dist-test-slave-pww0 krb5kdc[29693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:57:10 dist-test-slave-pww0 krb5kdc[29693](info): set up 2 sockets
Aug 08 03:57:10 dist-test-slave-pww0 krb5kdc[29693](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 03:57:11.700234 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:57:11 dist-test-slave-pww0 krb5kdc[29693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:57:11.911324 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:35107
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:32819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:35107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:12.265151 29716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:12.265484 29716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:12.265580 29716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:12.273129 29716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:12.273255 29716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:12.273309 29716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:12.273370 29716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:12.273415 29716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:57:12.283757 29716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:32819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:35107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:35107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:12.285137 29716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:12.286550 29716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:12.299243 29725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:12.299337 29723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:12.299374 29732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:12.299788 29716 server_base.cc:1034] running on GCE node
I20260808 03:57:12.300674 29716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:12.302242 29716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:12.303512 29716 hybrid_clock.cc:648] HybridClock initialized: now 1786161432303451 us; error 101 us; skew 500 ppm
I20260808 03:57:12.303912 29716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:12 dist-test-slave-pww0 krb5kdc[29693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161432, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:12.307161 29716 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:57:12.308734 29716 webserver.cc:460] Webserver started at http://127.28.251.254:38687/ using document root <none> and password file <none>
I20260808 03:57:12.309446 29716 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:12.309571 29716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:12.309883 29716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:12.313951 29716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "334fab8e042d4d7b9185aee6d96b0cc6"
format_stamp: "Formatted at 2026-08-08 03:57:12 on dist-test-slave-pww0"
server_key: "4de591c9ca35ffe0f125928204ce8610"
server_key_iv: "a7c22e9748ef157b00bf15717941b242"
server_key_version: "encryptionkey@0"
I20260808 03:57:12.314762 29716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "334fab8e042d4d7b9185aee6d96b0cc6"
format_stamp: "Formatted at 2026-08-08 03:57:12 on dist-test-slave-pww0"
server_key: "4de591c9ca35ffe0f125928204ce8610"
server_key_iv: "a7c22e9748ef157b00bf15717941b242"
server_key_version: "encryptionkey@0"
I20260808 03:57:12.320386 29716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 03:57:12.324306 29742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:12.325759 29716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:57:12.325941 29716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "334fab8e042d4d7b9185aee6d96b0cc6"
format_stamp: "Formatted at 2026-08-08 03:57:12 on dist-test-slave-pww0"
server_key: "4de591c9ca35ffe0f125928204ce8610"
server_key_iv: "a7c22e9748ef157b00bf15717941b242"
server_key_version: "encryptionkey@0"
I20260808 03:57:12.326148 29716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:12.348807 29716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:12.353355 29716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:12.353675 29716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:12.373268 29716 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:35107
I20260808 03:57:12.373317 29831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:35107 every 8 connection(s)
I20260808 03:57:12.374775 29716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:12.378858 29832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:12.384130 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 29716
I20260808 03:57:12.384498 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:12.387171 29679 external_mini_cluster.cc:1428] Setting key 67cfbbe3e01fd5cadb0fb8a82ee4ac3a
I20260808 03:57:12.392628 29832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6: Bootstrap starting.
I20260808 03:57:12.396190 29832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:12.397274 29832 log.cc:824] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:12.400235 29832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6: No bootstrap required, opened a new log
I20260808 03:57:12.409965 29832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334fab8e042d4d7b9185aee6d96b0cc6" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35107 } }
I20260808 03:57:12.410559 29832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:12.410688 29832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 334fab8e042d4d7b9185aee6d96b0cc6, State: Initialized, Role: FOLLOWER
I20260808 03:57:12.411485 29832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334fab8e042d4d7b9185aee6d96b0cc6" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35107 } }
I20260808 03:57:12.411782 29832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:12.411940 29832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:12.412148 29832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:12.414846 29832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334fab8e042d4d7b9185aee6d96b0cc6" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35107 } }
I20260808 03:57:12.415674 29832 leader_election.cc:302] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 334fab8e042d4d7b9185aee6d96b0cc6; no voters:
I20260808 03:57:12.416357 29832 leader_election.cc:288] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:12.416555 29840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:12.417701 29840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [term 1 LEADER]: Becoming Leader. State: Replica: 334fab8e042d4d7b9185aee6d96b0cc6, State: Running, Role: LEADER
I20260808 03:57:12.418454 29840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334fab8e042d4d7b9185aee6d96b0cc6" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35107 } }
I20260808 03:57:12.418675 29832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 03:57:12 dist-test-slave-pww0 krb5kdc[29693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:12.421867 29842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 334fab8e042d4d7b9185aee6d96b0cc6. Latest consensus state: current_term: 1 leader_uuid: "334fab8e042d4d7b9185aee6d96b0cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334fab8e042d4d7b9185aee6d96b0cc6" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35107 } } }
I20260808 03:57:12.422348 29842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:12.423002 29841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "334fab8e042d4d7b9185aee6d96b0cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334fab8e042d4d7b9185aee6d96b0cc6" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35107 } } }
I20260808 03:57:12.423363 29841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:12.437844 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:12.392569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.28.251.254:35107)
0808 03:57:12.393551 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:12.393567 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:12.394925 (+ 1358us) server_negotiation.cc:407] Connection header received
0808 03:57:12.405309 (+ 10384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:12.405344 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:12.405970 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:12.406489 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:12.409897 (+ 3408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:12.411961 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:12.414061 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:12.415191 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:12.415406 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:12.424419 (+ 9013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:12.424434 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:12.424459 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:12.424525 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:12.429515 (+ 4990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:12.431316 (+ 1801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:12.431328 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:12.431338 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:12.431664 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:12.432393 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:12.432399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:12.432403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:12.433328 (+ 925us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:12.433702 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:12.435311 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0808 03:57:12.435954 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":409,"threads_started":1}
I20260808 03:57:12.444448 29859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:12.449774 29859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:12.460631 29859 catalog_manager.cc:1347] Generated new cluster ID: b65aa723036945aaa896ab2d0f6ae91f
I20260808 03:57:12.460790 29859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:12.486217 29859 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:12.487931 29859 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:12.496886 29859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 334fab8e042d4d7b9185aee6d96b0cc6: Generated new TSK 0
I20260808 03:57:12.497886 29859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:57:12.559689 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35107
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:32819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:12.915498 29885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:12.915858 29885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:12.916009 29885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:12.923630 29885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:12.923776 29885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:12.923959 29885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:57:12.934499 29885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:32819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:12.935993 29885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:12.937463 29885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:12.952042 29890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:12.952001 29901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:12.951961 29896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:12.952718 29885 server_base.cc:1034] running on GCE node
I20260808 03:57:12.953424 29885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:12.954450 29885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:12.955667 29885 hybrid_clock.cc:648] HybridClock initialized: now 1786161432955598 us; error 60 us; skew 500 ppm
I20260808 03:57:12.956015 29885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:12 dist-test-slave-pww0 krb5kdc[29693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161432, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:12.959378 29885 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:57:12.961056 29885 webserver.cc:460] Webserver started at http://127.28.251.193:35041/ using document root <none> and password file <none>
I20260808 03:57:12.961795 29885 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:12.961925 29885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:12.962265 29885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:12.965679 29885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "3c772b87cdc3484c9295d7615d91bef7"
format_stamp: "Formatted at 2026-08-08 03:57:12 on dist-test-slave-pww0"
server_key: "3ee7e4b22f0007c751cb9f4caa282586"
server_key_iv: "76ab62266635996d4adcf2b622b08859"
server_key_version: "encryptionkey@0"
I20260808 03:57:12.966521 29885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "3c772b87cdc3484c9295d7615d91bef7"
format_stamp: "Formatted at 2026-08-08 03:57:12 on dist-test-slave-pww0"
server_key: "3ee7e4b22f0007c751cb9f4caa282586"
server_key_iv: "76ab62266635996d4adcf2b622b08859"
server_key_version: "encryptionkey@0"
I20260808 03:57:12.972285 29885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 03:57:12.976505 29912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:12.978113 29885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:57:12.978328 29885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "3c772b87cdc3484c9295d7615d91bef7"
format_stamp: "Formatted at 2026-08-08 03:57:12 on dist-test-slave-pww0"
server_key: "3ee7e4b22f0007c751cb9f4caa282586"
server_key_iv: "76ab62266635996d4adcf2b622b08859"
server_key_version: "encryptionkey@0"
I20260808 03:57:12.978559 29885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:12.997833 29885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:13.002703 29885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:13.003039 29885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:13.004245 29885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:13.005851 29885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:13.005945 29885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:13.006094 29885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:13.006152 29885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:13.040046 29885 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:38493
I20260808 03:57:13.040088 30110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:38493 every 8 connection(s)
I20260808 03:57:13.041431 29885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:57:13.042827 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 29885
I20260808 03:57:13.043115 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:57:13.043984 29679 external_mini_cluster.cc:1428] Setting key 14cdce98052a2ded7be1b56680020fac
Aug 08 03:57:13 dist-test-slave-pww0 krb5kdc[29693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161432, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
I20260808 03:57:13.063951 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:13.044658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:44745 (local address 127.28.251.254:35107)
0808 03:57:13.045273 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:13.045279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:13.046411 (+ 1132us) server_negotiation.cc:407] Connection header received
0808 03:57:13.048600 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:13.048606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:13.048741 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:13.048928 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:13.050594 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:13.051357 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:13.052996 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:13.053596 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:13.053735 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:13.057619 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:13.057625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:13.057628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:13.057667 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:13.060422 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:13.061297 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:13.061302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:13.061304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:13.061455 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:13.061951 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:13.061955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:13.061957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:13.062248 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:13.062391 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:13.063271 (+ 880us) server_negotiation.cc:299] Negotiation successful
0808 03:57:13.063558 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":358,"threads_started":1}
I20260808 03:57:13.065022 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:13.045332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:35107 (local address 127.28.251.193:44745)
0808 03:57:13.046240 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:13.046287 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:13.048300 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:13.049170 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:13.049180 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:13.049774 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:13.050408 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:13.050428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:13.051545 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:13.051549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:13.052850 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:13.052864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:13.053864 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:13.053870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:13.054101 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:13.054794 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:13.054818 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:13.057368 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:13.060615 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:13.060624 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:13.060641 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:13.061146 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:13.061578 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:13.061584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:13.061588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:13.061818 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:13.062393 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:13.062403 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:13.062801 (+ 398us) client_negotiation.cc:769] Sending connection context
0808 03:57:13.063132 (+ 331us) client_negotiation.cc:240] Negotiation successful
0808 03:57:13.063781 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":331,"threads_started":1}
I20260808 03:57:13.066737 30111 heartbeater.cc:347] Connected to a master server at 127.28.251.254:35107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
I20260808 03:57:13.067157 30111 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:13.068061 30111 heartbeater.cc:511] Master 127.28.251.254:35107 requested a full tablet report, sending...
I20260808 03:57:13.070689 29767 ts_manager.cc:194] Registered new tserver with Master: 3c772b87cdc3484c9295d7615d91bef7 (127.28.251.193:38493)
I20260808 03:57:13.072665 29767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:44745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:57:13.095250 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35107
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:32819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:13.454762 30126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:13.455118 30126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:13.455286 30126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:13.463006 30126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:13.463150 30126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:13.463307 30126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:57:13.474103 30126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:32819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:13.475600 30126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:13.477082 30126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:13.490823 30136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:13.490928 30138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:13.490888 30134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:13.491768 30126 server_base.cc:1034] running on GCE node
I20260808 03:57:13.492434 30126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:13.493444 30126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:13.494691 30126 hybrid_clock.cc:648] HybridClock initialized: now 1786161433494614 us; error 49 us; skew 500 ppm
I20260808 03:57:13.495066 30126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:13 dist-test-slave-pww0 krb5kdc[29693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161433, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:13.498521 30126 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:57:13.500177 30126 webserver.cc:460] Webserver started at http://127.28.251.194:36881/ using document root <none> and password file <none>
I20260808 03:57:13.500941 30126 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:13.501071 30126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:13.501402 30126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:13.504652 30126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "814bd1a0c85f423b9be2839b345c5faa"
format_stamp: "Formatted at 2026-08-08 03:57:13 on dist-test-slave-pww0"
server_key: "4d27f4533b1e37b837bd67f201ffb566"
server_key_iv: "381ceb90ff3d43bd387cf27f238a2a67"
server_key_version: "encryptionkey@0"
I20260808 03:57:13.505470 30126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "814bd1a0c85f423b9be2839b345c5faa"
format_stamp: "Formatted at 2026-08-08 03:57:13 on dist-test-slave-pww0"
server_key: "4d27f4533b1e37b837bd67f201ffb566"
server_key_iv: "381ceb90ff3d43bd387cf27f238a2a67"
server_key_version: "encryptionkey@0"
I20260808 03:57:13.511179 30126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 03:57:13.514964 30147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:13.516469 30126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:13.516637 30126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "814bd1a0c85f423b9be2839b345c5faa"
format_stamp: "Formatted at 2026-08-08 03:57:13 on dist-test-slave-pww0"
server_key: "4d27f4533b1e37b837bd67f201ffb566"
server_key_iv: "381ceb90ff3d43bd387cf27f238a2a67"
server_key_version: "encryptionkey@0"
I20260808 03:57:13.516868 30126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:13.540714 30126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:13.545459 30126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:13.545806 30126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:13.546911 30126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:13.548518 30126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:13.548596 30126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:13.548700 30126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:13.548753 30126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:13.581445 30126 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:36257
I20260808 03:57:13.581537 30341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:36257 every 8 connection(s)
I20260808 03:57:13.582901 30126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:57:13.587952 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 30126
I20260808 03:57:13.588150 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:57:13.588580 29679 external_mini_cluster.cc:1428] Setting key 670dde7911341d921d974dd82bd59f4c
Aug 08 03:57:13 dist-test-slave-pww0 krb5kdc[29693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161433, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:13.605595 30344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:13.585799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:47567 (local address 127.28.251.254:35107)
0808 03:57:13.586264 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:13.586269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:13.587381 (+ 1112us) server_negotiation.cc:407] Connection header received
0808 03:57:13.589551 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:13.589557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:13.589655 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:13.589770 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:13.591827 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:13.592567 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:13.594769 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:13.595208 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:13.595311 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:13.599542 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:13.599548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:13.599551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:13.599589 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:13.602167 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:13.602986 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:13.602990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:13.602993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:13.603128 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:13.603616 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:13.603620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:13.603622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:13.603924 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:13.604065 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:13.604912 (+ 847us) server_negotiation.cc:299] Negotiation successful
0808 03:57:13.605171 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":235,"threads_started":1}
I20260808 03:57:13.606401 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:13.586444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:35107 (local address 127.28.251.194:47567)
0808 03:57:13.587209 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:13.587249 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:13.589207 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:13.590010 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:13.590019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:13.590700 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:13.591608 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:13.591639 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:13.592762 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:13.592768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:13.594598 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:13.594614 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:13.595441 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:13.595448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:13.595778 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:13.596548 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:13.596578 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:13.599244 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:13.602341 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:13.602352 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:13.602371 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:13.602849 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:13.603248 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:13.603253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:13.603258 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:13.603494 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:13.604054 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:13.604063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:13.604447 (+ 384us) client_negotiation.cc:769] Sending connection context
0808 03:57:13.604743 (+ 296us) client_negotiation.cc:240] Negotiation successful
0808 03:57:13.605226 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":295,"threads_started":1}
I20260808 03:57:13.607990 30342 heartbeater.cc:347] Connected to a master server at 127.28.251.254:35107
I20260808 03:57:13.608397 30342 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
I20260808 03:57:13.609413 30342 heartbeater.cc:511] Master 127.28.251.254:35107 requested a full tablet report, sending...
I20260808 03:57:13.611121 29767 ts_manager.cc:194] Registered new tserver with Master: 814bd1a0c85f423b9be2839b345c5faa (127.28.251.194:36257)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
I20260808 03:57:13.612232 29767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:47567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:57:13.638605 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35107
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:32819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:13.998392 30349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:13.998744 30349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:13.998857 30349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:14.006708 30349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:14.006852 30349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:14.006978 30349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:57:14.017871 30349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:32819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:14.019341 30349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:14.020979 30349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:14.035337 30358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:14.035377 30360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:14.035305 30357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:14.036219 30349 server_base.cc:1034] running on GCE node
I20260808 03:57:14.036981 30349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:14.038080 30349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:14.039249 30349 hybrid_clock.cc:648] HybridClock initialized: now 1786161434039185 us; error 63 us; skew 500 ppm
I20260808 03:57:14.039688 30349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:14 dist-test-slave-pww0 krb5kdc[29693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161434, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:14.043166 30349 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:57:14.046032 30349 webserver.cc:460] Webserver started at http://127.28.251.195:42867/ using document root <none> and password file <none>
I20260808 03:57:14.046778 30349 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:14.046898 30349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:14.047245 30349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:14.050446 30349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "39b0da97f2c14103a6a96fcec79f0a3c"
format_stamp: "Formatted at 2026-08-08 03:57:14 on dist-test-slave-pww0"
server_key: "fb64796b5173ec105ffc50d27ecc08dc"
server_key_iv: "8601b715258bb421bc29d774df3e7cd0"
server_key_version: "encryptionkey@0"
I20260808 03:57:14.051304 30349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "39b0da97f2c14103a6a96fcec79f0a3c"
format_stamp: "Formatted at 2026-08-08 03:57:14 on dist-test-slave-pww0"
server_key: "fb64796b5173ec105ffc50d27ecc08dc"
server_key_iv: "8601b715258bb421bc29d774df3e7cd0"
server_key_version: "encryptionkey@0"
I20260808 03:57:14.057025 30349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 03:57:14.062381 30372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:14.064101 30349 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260808 03:57:14.064306 30349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "39b0da97f2c14103a6a96fcec79f0a3c"
format_stamp: "Formatted at 2026-08-08 03:57:14 on dist-test-slave-pww0"
server_key: "fb64796b5173ec105ffc50d27ecc08dc"
server_key_iv: "8601b715258bb421bc29d774df3e7cd0"
server_key_version: "encryptionkey@0"
I20260808 03:57:14.064550 30349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:14.076681 30111 heartbeater.cc:503] Master 127.28.251.254:35107 was elected leader, sending a full tablet report...
I20260808 03:57:14.094331 30349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:14.099030 30349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:14.099327 30349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:14.100412 30349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:14.101867 30349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:14.101950 30349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:14.102078 30349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:14.102137 30349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:14.135267 30349 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:41215
I20260808 03:57:14.135319 30555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:41215 every 8 connection(s)
I20260808 03:57:14.136646 30349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:57:14.141594 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 30349
I20260808 03:57:14.141777 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:57:14.142222 29679 external_mini_cluster.cc:1428] Setting key d14e53417b59c63a75d67af854e622f6
Aug 08 03:57:14 dist-test-slave-pww0 krb5kdc[29693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161434, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:14.158610 30559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:14.139530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:36361 (local address 127.28.251.254:35107)
0808 03:57:14.139991 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:14.139996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:14.141003 (+ 1007us) server_negotiation.cc:407] Connection header received
0808 03:57:14.143041 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:14.143046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:14.143121 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:14.143247 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:14.145229 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.146198 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.147928 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.148354 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.148497 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:14.152153 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:14.152159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:14.152162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:14.152203 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:14.154935 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.155856 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.155860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.155863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.156009 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.156466 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.156470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.156473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.156902 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:14.157077 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:14.157977 (+ 900us) server_negotiation.cc:299] Negotiation successful
0808 03:57:14.158228 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":226,"threads_started":1}
I20260808 03:57:14.159724 30561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:14.140054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:35107 (local address 127.28.251.195:36361)
0808 03:57:14.140827 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:14.140875 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:14.142738 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:14.143511 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:14.143523 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:14.144303 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:14.145024 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:14.145054 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.146380 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.146385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:14.147757 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:14.147778 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.148562 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.148567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:14.148779 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:14.149425 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:14.149450 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:14.151870 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:14.155120 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:14.155129 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:14.155144 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:14.155721 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:14.156096 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:14.156099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:14.156102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:14.156301 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:14.157050 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:14.157058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:14.157502 (+ 444us) client_negotiation.cc:769] Sending connection context
0808 03:57:14.157849 (+ 347us) client_negotiation.cc:240] Negotiation successful
0808 03:57:14.158359 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":306,"threads_started":1}
I20260808 03:57:14.161211 30557 heartbeater.cc:347] Connected to a master server at 127.28.251.254:35107
I20260808 03:57:14.161537 30557 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:14.162297 30557 heartbeater.cc:511] Master 127.28.251.254:35107 requested a full tablet report, sending...
I20260808 03:57:14.163977 29767 ts_manager.cc:194] Registered new tserver with Master: 39b0da97f2c14103a6a96fcec79f0a3c (127.28.251.195:41215)
I20260808 03:57:14.164835 29767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:36361
I20260808 03:57:14.171977 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 03:57:14 dist-test-slave-pww0 krb5kdc[29693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161434, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 03:57:14 dist-test-slave-pww0 krb5kdc[29693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161434, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:57:14.227246 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:14.210383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.28.251.193:38493)
0808 03:57:14.211208 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:14.211216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:14.211266 (+ 50us) server_negotiation.cc:407] Connection header received
0808 03:57:14.211335 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:14.211344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:14.211615 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:14.211751 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:14.213401 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.214487 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.215920 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.216648 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.216775 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:14.221083 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:14.221108 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:14.221131 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:14.221192 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:14.224313 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.225079 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.225086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.225088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.225250 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.225590 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.225594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.225596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.225995 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:14.226264 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:14.226593 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 03:57:14.226848 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":503,"threads_started":1}
W20260808 03:57:14.228631 29958 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46050
Aug 08 03:57:14 dist-test-slave-pww0 krb5kdc[29693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 03:57:14 dist-test-slave-pww0 krb5kdc[29693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:57:14.281471 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:14.266855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.28.251.193:38493)
0808 03:57:14.267097 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:14.267103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:14.267308 (+ 205us) server_negotiation.cc:407] Connection header received
0808 03:57:14.267502 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:14.267507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:14.267586 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:14.267678 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:14.269210 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.269986 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.271421 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.271943 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.272049 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:14.276084 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:14.276090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:14.276094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:14.276136 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:14.278767 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.279569 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.279573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.279576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.279716 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.280074 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.280077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.280079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.280352 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:14.280571 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:14.280928 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 03:57:14.281141 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 03:57:14.283170 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 29885
I20260808 03:57:14.296751 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:14.284317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.28.251.193:38493)
0808 03:57:14.284505 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:14.284511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:14.284790 (+ 279us) server_negotiation.cc:407] Connection header received
0808 03:57:14.284947 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:14.284952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:14.285012 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:14.285093 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:14.286396 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.287229 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.288559 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.288993 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.289092 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:14.291427 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:14.291432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:14.291436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:14.291529 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:14.294238 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.294914 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.294919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.294922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.295083 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.295412 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.295416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.295419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.295767 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:14.295958 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:14.296201 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 03:57:14.296378 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 03:57:14.297616 30102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:14.383450 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 29885
I20260808 03:57:14.407562 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 30126
Aug 08 03:57:14 dist-test-slave-pww0 krb5kdc[29693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 03:57:14.425107 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:14.408482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.28.251.194:36257)
0808 03:57:14.409041 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:14.409048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:14.409076 (+ 28us) server_negotiation.cc:407] Connection header received
0808 03:57:14.409138 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:14.409145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:14.409365 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:14.409516 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:14.410860 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.412180 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.413484 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.414066 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.414197 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:14.418460 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:14.418475 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:14.418491 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:14.418546 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:14.421958 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.422731 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.422739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.422742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.422894 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.423346 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.423350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.423352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.423786 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:14.424130 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:14.424425 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 03:57:14.424731 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
I20260808 03:57:14.426492 30337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:14.511574 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 30126
I20260808 03:57:14.533288 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 30349
Aug 08 03:57:14 dist-test-slave-pww0 krb5kdc[29693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 03:57:14.550384 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:14.534475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.28.251.195:41215)
0808 03:57:14.535162 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:14.535169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:14.535199 (+ 30us) server_negotiation.cc:407] Connection header received
0808 03:57:14.535268 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:14.535275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:14.535527 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:14.535678 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:14.537073 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.538256 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.539637 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:14.540270 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:14.540398 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:14.544770 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:14.544786 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:14.544803 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:14.544846 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:14.547813 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.548535 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.548542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.548544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.548673 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:14.548998 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:14.549001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:14.549003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:14.549366 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:14.549614 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:14.549859 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 03:57:14.550076 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":356,"threads_started":1}
I20260808 03:57:14.551826 30549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:14.636946 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 30349
I20260808 03:57:14.662581 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 29716
I20260808 03:57:14.664003 29824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:14.773058 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 29716
2026-08-08T03:57:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4369 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:57:14 dist-test-slave-pww0 krb5kdc[30599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:57:14 dist-test-slave-pww0 krb5kdc[30599](info): set up 2 sockets
Aug 08 03:57:14 dist-test-slave-pww0 krb5kdc[30599](info): commencing operation
krb5kdc: starting...
W20260808 03:57:16.009521 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:57:16 dist-test-slave-pww0 krb5kdc[30599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:57:16.235879 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:37167
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:37167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:16.579816 30621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:16.580137 30621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:16.580260 30621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:16.587697 30621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:16.587826 30621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:16.587901 30621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:16.587986 30621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:16.588052 30621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:57:16.598292 30621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:37167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:37167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:16.599732 30621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:16.601131 30621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:16.614213 30631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:16.614255 30628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:16.614288 30629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:16.614954 30621 server_base.cc:1034] running on GCE node
I20260808 03:57:16.615877 30621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:16.617425 30621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:16.618646 30621 hybrid_clock.cc:648] HybridClock initialized: now 1786161436618581 us; error 55 us; skew 500 ppm
I20260808 03:57:16.618986 30621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:16 dist-test-slave-pww0 krb5kdc[30599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161436, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:16.626963 30621 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:57:16.628683 30621 webserver.cc:460] Webserver started at http://127.28.251.254:38923/ using document root <none> and password file <none>
I20260808 03:57:16.629423 30621 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:16.629537 30621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:16.629834 30621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:16.633137 30621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "ad026c0af952451ea36267d5335e78c0"
format_stamp: "Formatted at 2026-08-08 03:57:16 on dist-test-slave-pww0"
server_key: "0d45aed0f8498d976e23fa4778e809f4"
server_key_iv: "0908b204be38ae29930c6a2505f7a521"
server_key_version: "encryptionkey@0"
I20260808 03:57:16.633929 30621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "ad026c0af952451ea36267d5335e78c0"
format_stamp: "Formatted at 2026-08-08 03:57:16 on dist-test-slave-pww0"
server_key: "0d45aed0f8498d976e23fa4778e809f4"
server_key_iv: "0908b204be38ae29930c6a2505f7a521"
server_key_version: "encryptionkey@0"
I20260808 03:57:16.639663 30621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 03:57:16.643647 30642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:16.645308 30621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:57:16.645501 30621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "ad026c0af952451ea36267d5335e78c0"
format_stamp: "Formatted at 2026-08-08 03:57:16 on dist-test-slave-pww0"
server_key: "0d45aed0f8498d976e23fa4778e809f4"
server_key_iv: "0908b204be38ae29930c6a2505f7a521"
server_key_version: "encryptionkey@0"
I20260808 03:57:16.645715 30621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:16.663762 30621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:16.668586 30621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:16.668900 30621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:16.688203 30621 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:37167
I20260808 03:57:16.688222 30723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:37167 every 8 connection(s)
I20260808 03:57:16.689692 30621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:16.693756 30724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:16.698824 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 30621
I20260808 03:57:16.699086 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:16.699846 29679 external_mini_cluster.cc:1428] Setting key 276f84fad263a7bd4409d06d52c223de
I20260808 03:57:16.707322 30724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0: Bootstrap starting.
I20260808 03:57:16.710918 30724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:16.712157 30724 log.cc:824] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:16.715642 30724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0: No bootstrap required, opened a new log
Aug 08 03:57:16 dist-test-slave-pww0 krb5kdc[30599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:16.724814 30724 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad026c0af952451ea36267d5335e78c0" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37167 } }
I20260808 03:57:16.725312 30724 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:16.725431 30724 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad026c0af952451ea36267d5335e78c0, State: Initialized, Role: FOLLOWER
I20260808 03:57:16.726179 30724 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad026c0af952451ea36267d5335e78c0" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37167 } }
I20260808 03:57:16.726511 30724 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:16.726679 30724 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:16.726898 30724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:16.728391 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:16.702457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.28.251.254:37167)
0808 03:57:16.705205 (+ 2748us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:16.705220 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:16.705876 (+ 656us) server_negotiation.cc:407] Connection header received
0808 03:57:16.707961 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:16.707988 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:16.708446 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:16.708856 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:16.710484 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:16.711989 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:16.713630 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:16.714352 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:16.714560 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:16.719543 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:16.719558 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:16.719577 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:16.719630 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:16.723112 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:16.724184 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:16.724193 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:16.724201 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:16.724404 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:16.724976 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:16.724982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:16.724985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:16.725688 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:16.726001 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:16.726612 (+ 611us) server_negotiation.cc:299] Negotiation successful
0808 03:57:16.727121 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2472,"thread_start_us":2187,"threads_started":1}
I20260808 03:57:16.729521 30724 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad026c0af952451ea36267d5335e78c0" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37167 } }
I20260808 03:57:16.730165 30724 leader_election.cc:302] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad026c0af952451ea36267d5335e78c0; no voters:
I20260808 03:57:16.730949 30724 leader_election.cc:288] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:16.731050 30734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:16.732684 30734 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [term 1 LEADER]: Becoming Leader. State: Replica: ad026c0af952451ea36267d5335e78c0, State: Running, Role: LEADER
I20260808 03:57:16.733310 30724 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:16.733255 30734 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad026c0af952451ea36267d5335e78c0" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37167 } }
I20260808 03:57:16.736088 30735 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad026c0af952451ea36267d5335e78c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad026c0af952451ea36267d5335e78c0" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37167 } } }
I20260808 03:57:16.736259 30736 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad026c0af952451ea36267d5335e78c0. Latest consensus state: current_term: 1 leader_uuid: "ad026c0af952451ea36267d5335e78c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad026c0af952451ea36267d5335e78c0" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37167 } } }
I20260808 03:57:16.736692 30735 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:16.736692 30736 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:16.738983 30753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:16.743834 30753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:16.753705 30753 catalog_manager.cc:1347] Generated new cluster ID: 8134205571604bb4a02a68fb96613d38
I20260808 03:57:16.753862 30753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:16.775620 30753 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:16.777269 30753 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:16.788620 30753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad026c0af952451ea36267d5335e78c0: Generated new TSK 0
I20260808 03:57:16.789585 30753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:57:16.856149 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37167
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:17.221026 30779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:17.221344 30779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:17.221475 30779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:17.229297 30779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:17.229429 30779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:17.229598 30779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:57:17.240598 30779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37167
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:17.242007 30779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:17.243517 30779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:17.258080 30792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:17.258121 30788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:17.258167 30787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:17.258896 30779 server_base.cc:1034] running on GCE node
I20260808 03:57:17.259694 30779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:17.260788 30779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:17.262017 30779 hybrid_clock.cc:648] HybridClock initialized: now 1786161437261957 us; error 47 us; skew 500 ppm
I20260808 03:57:17.262380 30779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:17 dist-test-slave-pww0 krb5kdc[30599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161437, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:17.265872 30779 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:57:17.267637 30779 webserver.cc:460] Webserver started at http://127.28.251.193:44979/ using document root <none> and password file <none>
I20260808 03:57:17.268393 30779 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:17.268538 30779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:17.268878 30779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:17.272351 30779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "0e93dfda4fd4423381a9722aef03e16f"
format_stamp: "Formatted at 2026-08-08 03:57:17 on dist-test-slave-pww0"
server_key: "4d4674923de65f0196a2bf2b68737a97"
server_key_iv: "20342de735e1a7134a096663ca71fd4f"
server_key_version: "encryptionkey@0"
I20260808 03:57:17.273291 30779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "0e93dfda4fd4423381a9722aef03e16f"
format_stamp: "Formatted at 2026-08-08 03:57:17 on dist-test-slave-pww0"
server_key: "4d4674923de65f0196a2bf2b68737a97"
server_key_iv: "20342de735e1a7134a096663ca71fd4f"
server_key_version: "encryptionkey@0"
I20260808 03:57:17.279048 30779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 03:57:17.282939 30802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:17.284473 30779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:17.284658 30779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "0e93dfda4fd4423381a9722aef03e16f"
format_stamp: "Formatted at 2026-08-08 03:57:17 on dist-test-slave-pww0"
server_key: "4d4674923de65f0196a2bf2b68737a97"
server_key_iv: "20342de735e1a7134a096663ca71fd4f"
server_key_version: "encryptionkey@0"
I20260808 03:57:17.284885 30779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:17.306644 30779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:17.311618 30779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:17.311908 30779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:17.312966 30779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:17.314415 30779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:17.314497 30779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:17.314585 30779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:17.314630 30779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:17.347931 30779 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:46237
I20260808 03:57:17.347939 30993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:46237 every 8 connection(s)
I20260808 03:57:17.349442 30779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:57:17.351120 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 30779
I20260808 03:57:17.351363 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:57:17.352128 29679 external_mini_cluster.cc:1428] Setting key 676c5eb817cc752bbc889501425950bd
Aug 08 03:57:17 dist-test-slave-pww0 krb5kdc[30599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161437, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:17.371981 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:17.352614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:50503 (local address 127.28.251.254:37167)
0808 03:57:17.353101 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:17.353107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:17.354394 (+ 1287us) server_negotiation.cc:407] Connection header received
0808 03:57:17.356483 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:17.356488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:17.356571 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:17.356668 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:17.358450 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:17.359215 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:17.360982 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:17.361399 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:17.361535 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:17.365613 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:17.365619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:17.365622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:17.365662 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:17.368462 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:17.369371 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:17.369376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:17.369378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:17.369517 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:17.370022 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:17.370025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:17.370027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:17.370315 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:17.370482 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:17.371292 (+ 810us) server_negotiation.cc:299] Negotiation successful
0808 03:57:17.371584 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":248,"threads_started":1}
I20260808 03:57:17.373040 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:17.353307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37167 (local address 127.28.251.193:50503)
0808 03:57:17.354219 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:17.354279 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:17.356155 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:17.356968 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:17.356978 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:17.357576 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:17.358239 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:17.358258 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:17.359405 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:17.359409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:17.360804 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:17.360820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:17.361681 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:17.361690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:17.361942 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:17.362716 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:17.362742 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:17.365300 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:17.368663 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:17.368672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:17.368689 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:17.369207 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:17.369654 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:17.369658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:17.369660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:17.369885 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:17.370483 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:17.370492 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:17.370877 (+ 385us) client_negotiation.cc:769] Sending connection context
0808 03:57:17.371133 (+ 256us) client_negotiation.cc:240] Negotiation successful
0808 03:57:17.371790 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":344,"threads_started":1}
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
I20260808 03:57:17.374724 30994 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37167
I20260808 03:57:17.375048 30994 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:17.375911 30994 heartbeater.cc:511] Master 127.28.251.254:37167 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
I20260808 03:57:17.378238 30672 ts_manager.cc:194] Registered new tserver with Master: 0e93dfda4fd4423381a9722aef03e16f (127.28.251.193:46237)
I20260808 03:57:17.379848 30672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:50503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:57:17.403224 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37167
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:17.757997 31003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:17.758325 31003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:17.758437 31003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:17.766085 31003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:17.766233 31003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:17.766360 31003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:57:17.777113 31003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37167
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:17.778496 31003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:17.779935 31003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:17.794462 31011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:17.794344 31013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:17.794410 31015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:17.795209 31003 server_base.cc:1034] running on GCE node
I20260808 03:57:17.795964 31003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:17.797130 31003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:17.798386 31003 hybrid_clock.cc:648] HybridClock initialized: now 1786161437798327 us; error 59 us; skew 500 ppm
I20260808 03:57:17.798779 31003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:17 dist-test-slave-pww0 krb5kdc[30599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161437, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:17.802198 31003 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:57:17.804124 31003 webserver.cc:460] Webserver started at http://127.28.251.194:41779/ using document root <none> and password file <none>
I20260808 03:57:17.804896 31003 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:17.805015 31003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:17.805326 31003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:17.808696 31003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "5a60fd6823cc4759bead8f4ff6817e5a"
format_stamp: "Formatted at 2026-08-08 03:57:17 on dist-test-slave-pww0"
server_key: "31685dc656efe05563b91d9f4b507a48"
server_key_iv: "9b97fe364f65f0708061f7c43dfc6222"
server_key_version: "encryptionkey@0"
I20260808 03:57:17.809563 31003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "5a60fd6823cc4759bead8f4ff6817e5a"
format_stamp: "Formatted at 2026-08-08 03:57:17 on dist-test-slave-pww0"
server_key: "31685dc656efe05563b91d9f4b507a48"
server_key_iv: "9b97fe364f65f0708061f7c43dfc6222"
server_key_version: "encryptionkey@0"
I20260808 03:57:17.815814 31003 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 03:57:17.820021 31029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:17.821708 31003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:57:17.821887 31003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "5a60fd6823cc4759bead8f4ff6817e5a"
format_stamp: "Formatted at 2026-08-08 03:57:17 on dist-test-slave-pww0"
server_key: "31685dc656efe05563b91d9f4b507a48"
server_key_iv: "9b97fe364f65f0708061f7c43dfc6222"
server_key_version: "encryptionkey@0"
I20260808 03:57:17.822115 31003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:17.841773 31003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:17.846490 31003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:17.846799 31003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:17.847965 31003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:17.849572 31003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:17.849653 31003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:17.849774 31003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:17.849826 31003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:17.883327 31003 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:34959
I20260808 03:57:17.883359 31225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:34959 every 8 connection(s)
I20260808 03:57:17.884845 31003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:57:17.886901 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 31003
I20260808 03:57:17.887125 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:57:17.887771 29679 external_mini_cluster.cc:1428] Setting key 1b4277ec7cc5ca7f499337b5617a5062
Aug 08 03:57:17 dist-test-slave-pww0 krb5kdc[30599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161437, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:17.909101 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:17.887798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:46047 (local address 127.28.251.254:37167)
0808 03:57:17.888272 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:17.888277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:17.889691 (+ 1414us) server_negotiation.cc:407] Connection header received
0808 03:57:17.891784 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:17.891790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:17.891879 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:17.892003 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:17.894098 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:17.895102 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:17.896906 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:17.897489 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:17.897622 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:17.902359 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:17.902365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:17.902369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:17.902418 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:17.905129 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:17.906352 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:17.906358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:17.906362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:17.906563 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:17.907146 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:17.907150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:17.907152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:17.907479 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:17.907612 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:17.908612 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0808 03:57:17.908823 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":237,"threads_started":1}
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
I20260808 03:57:17.910876 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:17.888509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37167 (local address 127.28.251.194:46047)
0808 03:57:17.889481 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:17.889552 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:17.891438 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:17.892403 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:17.892417 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:17.893231 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:17.893901 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:17.893924 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:17.895301 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:17.895305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:17.896723 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:17.896745 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:17.897807 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:17.897813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:17.898045 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:17.898921 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:17.898961 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:17.902017 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:17.905340 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:17.905351 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:17.905372 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:17.906146 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:17.906707 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:17.906713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:17.906716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:17.907017 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:17.907623 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:17.907633 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:17.908176 (+ 543us) client_negotiation.cc:769] Sending connection context
0808 03:57:17.908523 (+ 347us) client_negotiation.cc:240] Negotiation successful
0808 03:57:17.909166 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":378,"threads_started":1}
I20260808 03:57:17.912539 31226 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37167
I20260808 03:57:17.912882 31226 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:17.913659 31226 heartbeater.cc:511] Master 127.28.251.254:37167 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
I20260808 03:57:17.915444 30672 ts_manager.cc:194] Registered new tserver with Master: 5a60fd6823cc4759bead8f4ff6817e5a (127.28.251.194:34959)
I20260808 03:57:17.916386 30672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:46047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:57:17.940675 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37167
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:18.304196 31236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:18.304551 31236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:18.304729 31236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:18.312528 31236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:18.312696 31236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:18.312875 31236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:57:18.323765 31236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37167
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:18.325316 31236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:18.326958 31236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:18.341604 31243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:18.341671 31242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:18.341742 31246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:18.342417 31236 server_base.cc:1034] running on GCE node
I20260808 03:57:18.343199 31236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:18.344383 31236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:18.345650 31236 hybrid_clock.cc:648] HybridClock initialized: now 1786161438345581 us; error 52 us; skew 500 ppm
I20260808 03:57:18.346119 31236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:18 dist-test-slave-pww0 krb5kdc[30599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161438, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:18.350289 31236 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:57:18.352103 31236 webserver.cc:460] Webserver started at http://127.28.251.195:35723/ using document root <none> and password file <none>
I20260808 03:57:18.352885 31236 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:18.353017 31236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:18.353361 31236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:18.356594 31236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "f53acd0d0b894f37885a28667b738178"
format_stamp: "Formatted at 2026-08-08 03:57:18 on dist-test-slave-pww0"
server_key: "cec67c2416e3396ca1a974f22106c8a8"
server_key_iv: "bd563fbffeb4f35fd03fd82cae834c8d"
server_key_version: "encryptionkey@0"
I20260808 03:57:18.357453 31236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "f53acd0d0b894f37885a28667b738178"
format_stamp: "Formatted at 2026-08-08 03:57:18 on dist-test-slave-pww0"
server_key: "cec67c2416e3396ca1a974f22106c8a8"
server_key_iv: "bd563fbffeb4f35fd03fd82cae834c8d"
server_key_version: "encryptionkey@0"
I20260808 03:57:18.363072 31236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 03:57:18.367019 31259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:18.368549 31236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:18.368734 31236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "f53acd0d0b894f37885a28667b738178"
format_stamp: "Formatted at 2026-08-08 03:57:18 on dist-test-slave-pww0"
server_key: "cec67c2416e3396ca1a974f22106c8a8"
server_key_iv: "bd563fbffeb4f35fd03fd82cae834c8d"
server_key_version: "encryptionkey@0"
I20260808 03:57:18.368952 31236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:18.383062 30994 heartbeater.cc:503] Master 127.28.251.254:37167 was elected leader, sending a full tablet report...
I20260808 03:57:18.385567 31236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:18.390478 31236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:18.390784 31236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:18.391894 31236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:18.393404 31236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:18.393501 31236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:18.393639 31236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:18.393697 31236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:18.427861 31236 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:35017
I20260808 03:57:18.427906 31450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:35017 every 8 connection(s)
I20260808 03:57:18.429221 31236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:57:18.434711 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 31236
I20260808 03:57:18.434907 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:57:18.435350 29679 external_mini_cluster.cc:1428] Setting key e4ec560e3cc913468b835ed80b2ce282
Aug 08 03:57:18 dist-test-slave-pww0 krb5kdc[30599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161438, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:18.451550 31455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.431991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:53959 (local address 127.28.251.254:37167)
0808 03:57:18.432528 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.432535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.433606 (+ 1071us) server_negotiation.cc:407] Connection header received
0808 03:57:18.435752 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.435758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.435911 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.436032 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:18.438212 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.439201 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.440880 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.441293 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.441398 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.444994 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:18.445000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:18.445004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:18.445041 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:18.447854 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.448626 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.448630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.448632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.448768 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.449318 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.449322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.449324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.449730 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:18.449894 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:18.450869 (+ 975us) server_negotiation.cc:299] Negotiation successful
0808 03:57:18.451104 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":240,"threads_started":1}
I20260808 03:57:18.452632 31456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.432529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37167 (local address 127.28.251.195:53959)
0808 03:57:18.433412 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:18.433464 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:18.435381 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:18.436363 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:18.436376 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:18.437200 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:18.438010 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.438032 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.439368 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.439373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.440723 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.440737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.441506 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.441511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.441732 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.442423 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:18.442449 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:18.444751 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:18.448002 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:18.448010 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:18.448026 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:18.448504 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:18.448887 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:18.448891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:18.448893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:18.449184 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:18.449884 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:18.449893 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:18.450335 (+ 442us) client_negotiation.cc:769] Sending connection context
0808 03:57:18.450719 (+ 384us) client_negotiation.cc:240] Negotiation successful
0808 03:57:18.451341 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":351,"threads_started":1}
I20260808 03:57:18.454197 31452 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37167
I20260808 03:57:18.454556 31452 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:18.455346 31452 heartbeater.cc:511] Master 127.28.251.254:37167 requested a full tablet report, sending...
I20260808 03:57:18.457269 30672 ts_manager.cc:194] Registered new tserver with Master: f53acd0d0b894f37885a28667b738178 (127.28.251.195:35017)
I20260808 03:57:18.458173 30672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:53959
I20260808 03:57:18.465318 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 03:57:18.486253 31455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.472148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.28.251.254:37167)
0808 03:57:18.472384 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.472392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.472876 (+ 484us) server_negotiation.cc:407] Connection header received
0808 03:57:18.473048 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.473052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.473139 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.473265 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:18.474951 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.475878 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.477271 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.477748 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.477859 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.480497 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:18.480503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:18.480507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:18.480548 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:18.483266 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.484062 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.484067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.484069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.484218 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.484622 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.484627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.484629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.485023 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:18.485245 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:18.485684 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 03:57:18.485901 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 03:57:18.493537 30672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:57:18.500881 30672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:57:18.534458 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.514978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.28.251.193:46237)
0808 03:57:18.524310 (+ 9332us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.524319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.525444 (+ 1125us) server_negotiation.cc:407] Connection header received
0808 03:57:18.525589 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.525598 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.525994 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.526246 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:18.527161 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.528587 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.531597 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.532965 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.533175 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.533379 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:18.533938 (+ 559us) server_negotiation.cc:299] Negotiation successful
0808 03:57:18.534115 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9140,"thread_start_us":8978,"threads_started":1}
I20260808 03:57:18.534574 31481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.514180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46237 (local address 127.0.0.1:36742)
0808 03:57:18.525015 (+ 10835us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:18.525037 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:18.525209 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:18.526360 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:18.526370 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:18.526405 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:18.526930 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.526947 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.528777 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.528782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.531409 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.531424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.533523 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.533530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.533704 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.533734 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 03:57:18.533807 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 03:57:18.534020 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10704,"mutex_wait_us":53,"thread_start_us":10556,"threads_started":1}
I20260808 03:57:18.534876 31480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.514452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:34959 (local address 127.0.0.1:39494)
0808 03:57:18.525368 (+ 10916us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:18.525384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:18.525500 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:18.526671 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:18.526677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:18.526767 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:18.527242 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.527256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.529407 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.529413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.531245 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.531257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.533252 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.533260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.533473 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.533539 (+ 66us) client_negotiation.cc:769] Sending connection context
0808 03:57:18.533695 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 03:57:18.534390 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10788,"mutex_wait_us":438,"thread_start_us":8329,"threads_started":1}
I20260808 03:57:18.535439 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.513946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:35017 (local address 127.0.0.1:33760)
0808 03:57:18.523256 (+ 9310us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:18.523315 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:18.523559 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:18.527779 (+ 4220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:18.527784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:18.527813 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:18.528134 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.528146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.530092 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.530098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.532061 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.532073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.534066 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.534072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.534259 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.534288 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 03:57:18.534437 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 03:57:18.534583 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9002,"thread_start_us":11234,"threads_started":1}
I20260808 03:57:18.536064 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.514273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.28.251.195:35017)
0808 03:57:18.527138 (+ 12865us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.527147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.527194 (+ 47us) server_negotiation.cc:407] Connection header received
0808 03:57:18.527302 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.527309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.527567 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.527774 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:18.528349 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.529878 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.532231 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.533566 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.533749 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.533919 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:18.535600 (+ 1681us) server_negotiation.cc:299] Negotiation successful
0808 03:57:18.535773 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12664,"thread_start_us":12404,"threads_started":1}
I20260808 03:57:18.536339 31485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.514898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.28.251.194:34959)
0808 03:57:18.526039 (+ 11141us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.526048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.526077 (+ 29us) server_negotiation.cc:407] Connection header received
0808 03:57:18.526182 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.526189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.526434 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.527435 (+ 1001us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:18.527723 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.529238 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.531425 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.532965 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.534015 (+ 1050us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.534200 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:18.534267 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 03:57:18.535979 (+ 1712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10969,"thread_start_us":10901,"threads_started":1}
I20260808 03:57:18.539060 30877 tablet_service.cc:1467] Processing CreateTablet for tablet 520e0981d87c43ec8b993b98f82c0f1f (DEFAULT_TABLE table=test-table [id=c9fc3e88f7364685b300a8e003ed7677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:18.540457 30877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520e0981d87c43ec8b993b98f82c0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:18.540073 31110 tablet_service.cc:1467] Processing CreateTablet for tablet 520e0981d87c43ec8b993b98f82c0f1f (DEFAULT_TABLE table=test-table [id=c9fc3e88f7364685b300a8e003ed7677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:18.540073 31349 tablet_service.cc:1467] Processing CreateTablet for tablet 520e0981d87c43ec8b993b98f82c0f1f (DEFAULT_TABLE table=test-table [id=c9fc3e88f7364685b300a8e003ed7677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:18.541551 31349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520e0981d87c43ec8b993b98f82c0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:18.541545 31110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520e0981d87c43ec8b993b98f82c0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:18.552985 31487 tablet_bootstrap.cc:492] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f: Bootstrap starting.
I20260808 03:57:18.556838 31487 tablet_bootstrap.cc:654] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:18.558032 31491 tablet_bootstrap.cc:492] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178: Bootstrap starting.
I20260808 03:57:18.558445 31487 log.cc:824] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:18.559026 31488 tablet_bootstrap.cc:492] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a: Bootstrap starting.
I20260808 03:57:18.561568 31487 tablet_bootstrap.cc:492] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f: No bootstrap required, opened a new log
I20260808 03:57:18.561863 31487 ts_tablet_manager.cc:1397] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 03:57:18.563016 31491 tablet_bootstrap.cc:654] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:18.563014 31488 tablet_bootstrap.cc:654] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:18.564392 31488 log.cc:824] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:18.564385 31491 log.cc:824] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:18.569291 31488 tablet_bootstrap.cc:492] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a: No bootstrap required, opened a new log
I20260808 03:57:18.569443 31491 tablet_bootstrap.cc:492] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178: No bootstrap required, opened a new log
I20260808 03:57:18.569620 31488 ts_tablet_manager.cc:1397] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260808 03:57:18.569753 31491 ts_tablet_manager.cc:1397] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260808 03:57:18.571447 31487 raft_consensus.cc:348] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53acd0d0b894f37885a28667b738178" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35017 } } peers { permanent_uuid: "0e93dfda4fd4423381a9722aef03e16f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46237 } } peers { permanent_uuid: "5a60fd6823cc4759bead8f4ff6817e5a" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 34959 } }
I20260808 03:57:18.571975 31487 raft_consensus.cc:374] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:18.572114 31487 raft_consensus.cc:729] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e93dfda4fd4423381a9722aef03e16f, State: Initialized, Role: FOLLOWER
I20260808 03:57:18.572834 31487 consensus_queue.cc:260] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53acd0d0b894f37885a28667b738178" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35017 } } peers { permanent_uuid: "0e93dfda4fd4423381a9722aef03e16f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46237 } } peers { permanent_uuid: "5a60fd6823cc4759bead8f4ff6817e5a" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 34959 } }
I20260808 03:57:18.574970 31487 ts_tablet_manager.cc:1428] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260808 03:57:18.578701 31488 raft_consensus.cc:348] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53acd0d0b894f37885a28667b738178" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35017 } } peers { permanent_uuid: "0e93dfda4fd4423381a9722aef03e16f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46237 } } peers { permanent_uuid: "5a60fd6823cc4759bead8f4ff6817e5a" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 34959 } }
I20260808 03:57:18.579339 31488 raft_consensus.cc:374] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:18.579520 31488 raft_consensus.cc:729] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60fd6823cc4759bead8f4ff6817e5a, State: Initialized, Role: FOLLOWER
I20260808 03:57:18.580212 31488 consensus_queue.cc:260] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53acd0d0b894f37885a28667b738178" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35017 } } peers { permanent_uuid: "0e93dfda4fd4423381a9722aef03e16f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46237 } } peers { permanent_uuid: "5a60fd6823cc4759bead8f4ff6817e5a" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 34959 } }
I20260808 03:57:18.580190 31491 raft_consensus.cc:348] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53acd0d0b894f37885a28667b738178" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35017 } } peers { permanent_uuid: "0e93dfda4fd4423381a9722aef03e16f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46237 } } peers { permanent_uuid: "5a60fd6823cc4759bead8f4ff6817e5a" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 34959 } }
I20260808 03:57:18.580675 31491 raft_consensus.cc:374] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:18.580837 31491 raft_consensus.cc:729] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53acd0d0b894f37885a28667b738178, State: Initialized, Role: FOLLOWER
I20260808 03:57:18.581492 31226 heartbeater.cc:503] Master 127.28.251.254:37167 was elected leader, sending a full tablet report...
I20260808 03:57:18.581698 31491 consensus_queue.cc:260] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53acd0d0b894f37885a28667b738178" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35017 } } peers { permanent_uuid: "0e93dfda4fd4423381a9722aef03e16f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46237 } } peers { permanent_uuid: "5a60fd6823cc4759bead8f4ff6817e5a" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 34959 } }
I20260808 03:57:18.582319 31488 ts_tablet_manager.cc:1428] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260808 03:57:18.582886 31452 heartbeater.cc:503] Master 127.28.251.254:37167 was elected leader, sending a full tablet report...
I20260808 03:57:18.583549 31491 ts_tablet_manager.cc:1428] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260808 03:57:18.602504 30995 tablet.cc:2369] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 03:57:18.637072 31228 tablet.cc:2369] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:18.665656 31498 raft_consensus.cc:482] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:18.665980 31498 raft_consensus.cc:504] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53acd0d0b894f37885a28667b738178" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35017 } } peers { permanent_uuid: "0e93dfda4fd4423381a9722aef03e16f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46237 } } peers { permanent_uuid: "5a60fd6823cc4759bead8f4ff6817e5a" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 34959 } }
I20260808 03:57:18.667915 31498 leader_election.cc:288] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f53acd0d0b894f37885a28667b738178 (127.28.251.195:35017), 5a60fd6823cc4759bead8f4ff6817e5a (127.28.251.194:34959)
I20260808 03:57:18.676772 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.668544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:53155 (local address 127.28.251.195:35017)
0808 03:57:18.668764 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.668770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.669415 (+ 645us) server_negotiation.cc:407] Connection header received
0808 03:57:18.669571 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.669577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.669683 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.669768 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:18.670653 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.671809 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.674906 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.675858 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.676001 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.676059 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:18.676377 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 03:57:18.676513 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 03:57:18.676894 31504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.668455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:35017 (local address 127.28.251.193:53155)
0808 03:57:18.669285 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:18.669300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:18.669421 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:18.669803 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:18.669808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:18.669842 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:18.670415 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.670435 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.671941 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.671945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.674669 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.674687 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.676067 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.676072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.676189 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.676219 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 03:57:18.676313 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 03:57:18.676453 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":233,"spinlock_wait_cycles":36096,"thread_start_us":495,"threads_started":1}
I20260808 03:57:18.677939 31485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.668505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:44537 (local address 127.28.251.194:34959)
0808 03:57:18.668876 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.668882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.669350 (+ 468us) server_negotiation.cc:407] Connection header received
0808 03:57:18.669571 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.669577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.669662 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.669769 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:18.670631 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.671791 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.676146 (+ 4355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.677017 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.677182 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.677255 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:18.677585 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 03:57:18.677727 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260808 03:57:18.677996 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.668318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:34959 (local address 127.28.251.193:44537)
0808 03:57:18.669200 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:18.669231 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:18.669416 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:18.669803 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:18.669808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:18.669852 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:18.670489 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.670499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.671937 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.671943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.674677 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:18.674687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.677229 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.677234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:18.677365 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.677389 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 03:57:18.677516 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 03:57:18.677616 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":227,"spinlock_wait_cycles":4352,"thread_start_us":456,"threads_started":1}
I20260808 03:57:18.678364 31379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520e0981d87c43ec8b993b98f82c0f1f" candidate_uuid: "0e93dfda4fd4423381a9722aef03e16f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53acd0d0b894f37885a28667b738178" is_pre_election: true
I20260808 03:57:18.678908 31379 raft_consensus.cc:2454] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e93dfda4fd4423381a9722aef03e16f in term 0.
I20260808 03:57:18.679071 31145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520e0981d87c43ec8b993b98f82c0f1f" candidate_uuid: "0e93dfda4fd4423381a9722aef03e16f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60fd6823cc4759bead8f4ff6817e5a" is_pre_election: true
I20260808 03:57:18.679625 31145 raft_consensus.cc:2454] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e93dfda4fd4423381a9722aef03e16f in term 0.
I20260808 03:57:18.679642 30806 leader_election.cc:302] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e93dfda4fd4423381a9722aef03e16f, f53acd0d0b894f37885a28667b738178; no voters:
I20260808 03:57:18.680083 31498 raft_consensus.cc:2781] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:57:18.680234 31498 raft_consensus.cc:482] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:57:18.680343 31498 raft_consensus.cc:3037] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [term 0 FOLLOWER]: Advancing to term 1
W20260808 03:57:18.680856 31453 tablet.cc:2369] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:18.682647 31498 raft_consensus.cc:504] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53acd0d0b894f37885a28667b738178" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35017 } } peers { permanent_uuid: "0e93dfda4fd4423381a9722aef03e16f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46237 } } peers { permanent_uuid: "5a60fd6823cc4759bead8f4ff6817e5a" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 34959 } }
I20260808 03:57:18.683355 31498 leader_election.cc:288] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [CANDIDATE]: Term 1 election: Requested vote from peers f53acd0d0b894f37885a28667b738178 (127.28.251.195:35017), 5a60fd6823cc4759bead8f4ff6817e5a (127.28.251.194:34959)
I20260808 03:57:18.683912 31145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520e0981d87c43ec8b993b98f82c0f1f" candidate_uuid: "0e93dfda4fd4423381a9722aef03e16f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60fd6823cc4759bead8f4ff6817e5a"
I20260808 03:57:18.683912 31379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520e0981d87c43ec8b993b98f82c0f1f" candidate_uuid: "0e93dfda4fd4423381a9722aef03e16f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53acd0d0b894f37885a28667b738178"
I20260808 03:57:18.684139 31145 raft_consensus.cc:3037] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:18.684139 31379 raft_consensus.cc:3037] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:18.687405 31145 raft_consensus.cc:2454] T 520e0981d87c43ec8b993b98f82c0f1f P 5a60fd6823cc4759bead8f4ff6817e5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e93dfda4fd4423381a9722aef03e16f in term 1.
I20260808 03:57:18.687405 31379 raft_consensus.cc:2454] T 520e0981d87c43ec8b993b98f82c0f1f P f53acd0d0b894f37885a28667b738178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e93dfda4fd4423381a9722aef03e16f in term 1.
I20260808 03:57:18.687886 30806 leader_election.cc:302] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e93dfda4fd4423381a9722aef03e16f, f53acd0d0b894f37885a28667b738178; no voters:
I20260808 03:57:18.688284 31498 raft_consensus.cc:2781] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:18.688830 31498 raft_consensus.cc:686] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [term 1 LEADER]: Becoming Leader. State: Replica: 0e93dfda4fd4423381a9722aef03e16f, State: Running, Role: LEADER
I20260808 03:57:18.689432 31498 consensus_queue.cc:237] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53acd0d0b894f37885a28667b738178" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35017 } } peers { permanent_uuid: "0e93dfda4fd4423381a9722aef03e16f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46237 } } peers { permanent_uuid: "5a60fd6823cc4759bead8f4ff6817e5a" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 34959 } }
I20260808 03:57:18.694516 30672 catalog_manager.cc:5495] T 520e0981d87c43ec8b993b98f82c0f1f P 0e93dfda4fd4423381a9722aef03e16f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e93dfda4fd4423381a9722aef03e16f (127.28.251.193). New cstate: current_term: 1 leader_uuid: "0e93dfda4fd4423381a9722aef03e16f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53acd0d0b894f37885a28667b738178" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e93dfda4fd4423381a9722aef03e16f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60fd6823cc4759bead8f4ff6817e5a" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 34959 } health_report { overall_health: UNKNOWN } } }
Aug 08 03:57:18 dist-test-slave-pww0 krb5kdc[30599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:57:18.743268 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.727682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.28.251.193:46237)
0808 03:57:18.727938 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.727945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.728355 (+ 410us) server_negotiation.cc:407] Connection header received
0808 03:57:18.728537 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.728541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.728613 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.728716 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:18.730291 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.731050 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.732412 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.732835 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.732951 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.737184 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:18.737203 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:18.737221 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:18.737264 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:18.740535 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.741259 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.741269 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.741271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.741468 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.741794 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.741797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.741799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.742193 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:18.742463 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:18.742655 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 03:57:18.742863 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Aug 08 03:57:18 dist-test-slave-pww0 krb5kdc[30599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161438, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 03:57:18 dist-test-slave-pww0 krb5kdc[30599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161438, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:57:18.797885 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.783749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36762 (local address 127.28.251.193:46237)
0808 03:57:18.783967 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.783975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.784399 (+ 424us) server_negotiation.cc:407] Connection header received
0808 03:57:18.784583 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.784587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.784656 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.784745 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:18.786173 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.786922 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.788281 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.788803 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.788924 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.792914 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:18.792919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:18.792923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:18.792961 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:18.795411 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.796101 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.796104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.796107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.796271 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.796637 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.796640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.796642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.796930 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:18.797163 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:18.797370 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 03:57:18.797560 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 03:57:18 dist-test-slave-pww0 krb5kdc[30599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 03:57:18 dist-test-slave-pww0 krb5kdc[30599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:57:18.851536 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.836973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.28.251.193:46237)
0808 03:57:18.837176 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.837182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.837552 (+ 370us) server_negotiation.cc:407] Connection header received
0808 03:57:18.837735 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.837740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.837829 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.837928 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:18.839410 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.840240 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.841529 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.841962 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.842079 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.846181 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:18.846186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:18.846189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:18.846229 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:18.848865 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.849585 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.849590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.849592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.849741 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.850180 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.850184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.850186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.850524 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:18.850798 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:18.851008 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 03:57:18.851220 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 03:57:18.853334 30848 tablet_service.cc:2908] Rejecting scan request for tablet 520e0981d87c43ec8b993b98f82c0f1f: Uninitialized: clean time has not yet been initialized
I20260808 03:57:18.856861 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 30779
I20260808 03:57:18.870195 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.858145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.28.251.193:46237)
0808 03:57:18.858323 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.858327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.858710 (+ 383us) server_negotiation.cc:407] Connection header received
0808 03:57:18.858829 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.858833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.858898 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.858984 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:18.860226 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.860957 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.862132 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:18.862578 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:18.862676 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:18.864905 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:18.864910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:18.864913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:18.864946 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:18.867633 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.868344 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.868349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.868351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.868500 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:18.868847 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:18.868852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:18.868854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:18.869173 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:18.869374 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:18.869665 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 03:57:18.869900 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 03:57:18.870919 30987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:18.964699 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 30779
I20260808 03:57:18.997437 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31003
Aug 08 03:57:18 dist-test-slave-pww0 krb5kdc[30599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 03:57:19.012513 31485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:18.998422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.28.251.194:34959)
0808 03:57:18.998609 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:18.998614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:18.998632 (+ 18us) server_negotiation.cc:407] Connection header received
0808 03:57:18.998694 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:18.998697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:18.998775 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:18.998889 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:19.000150 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:19.000952 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:19.002244 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:19.002641 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:19.002737 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:19.006799 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:19.006819 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:19.006845 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:19.006887 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:19.009871 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:19.010572 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:19.010579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:19.010582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:19.010714 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:19.011032 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:19.011036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:19.011038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:19.011435 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:19.011884 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:19.012058 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 03:57:19.012242 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 03:57:19.013311 31216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:19.104272 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 31003
I20260808 03:57:19.133792 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31236
Aug 08 03:57:19 dist-test-slave-pww0 krb5kdc[30599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 03:57:19.150192 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:19.134744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.28.251.195:35017)
0808 03:57:19.134922 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:19.134926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:19.134947 (+ 21us) server_negotiation.cc:407] Connection header received
0808 03:57:19.135086 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:19.135090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:19.135174 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:19.135255 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:19.136681 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:19.137523 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:19.138969 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:19.139378 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:19.139521 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:19.143968 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:19.143989 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:19.144007 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:19.144063 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:19.146933 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:19.147870 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:19.147880 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:19.147882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:19.148057 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:19.148535 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:19.148539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:19.148541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:19.148961 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:19.149208 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:19.149664 (+ 456us) server_negotiation.cc:299] Negotiation successful
0808 03:57:19.149914 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 03:57:19.150957 31444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:19.240918 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 31236
I20260808 03:57:19.267768 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 30621
I20260808 03:57:19.269033 30717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:19.382126 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 30621
2026-08-08T03:57:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4610 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:57:19 dist-test-slave-pww0 krb5kdc[31545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:57:19 dist-test-slave-pww0 krb5kdc[31545](info): set up 2 sockets
Aug 08 03:57:19 dist-test-slave-pww0 krb5kdc[31545](info): commencing operation
krb5kdc: starting...
W20260808 03:57:20.638386 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:57:20 dist-test-slave-pww0 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:57:20.876310 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:42009
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:42009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:21.236475 31561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:21.236920 31561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:21.237138 31561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:21.246884 31561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:21.247025 31561 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 03:57:21.247118 31561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:21.247210 31561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:21.247282 31561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:21.247371 31561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:57:21.262223 31561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:42009
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:42009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:21.264034 31561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:21.265848 31561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:21.282250 31569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:21.283138 31568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:21.283433 31572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:21.283788 31561 server_base.cc:1034] running on GCE node
I20260808 03:57:21.285004 31561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:21.286656 31561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:21.287901 31561 hybrid_clock.cc:648] HybridClock initialized: now 1786161441287838 us; error 45 us; skew 500 ppm
I20260808 03:57:21.288300 31561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:21 dist-test-slave-pww0 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161441, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:21.291797 31561 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:57:21.293491 31561 webserver.cc:460] Webserver started at http://127.28.251.254:44927/ using document root <none> and password file <none>
I20260808 03:57:21.294251 31561 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:21.294358 31561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:21.294705 31561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:21.297931 31561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "c5ad389c0e0247df9a19715375ba140d"
format_stamp: "Formatted at 2026-08-08 03:57:21 on dist-test-slave-pww0"
server_key: "c2a870ef70b1828c8734937b7ffeddc7"
server_key_iv: "4869560f4704cc5e95f7d6b74fa7f756"
server_key_version: "encryptionkey@0"
I20260808 03:57:21.298784 31561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "c5ad389c0e0247df9a19715375ba140d"
format_stamp: "Formatted at 2026-08-08 03:57:21 on dist-test-slave-pww0"
server_key: "c2a870ef70b1828c8734937b7ffeddc7"
server_key_iv: "4869560f4704cc5e95f7d6b74fa7f756"
server_key_version: "encryptionkey@0"
I20260808 03:57:21.304920 31561 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260808 03:57:21.309159 31582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:21.310778 31561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 03:57:21.310997 31561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "c5ad389c0e0247df9a19715375ba140d"
format_stamp: "Formatted at 2026-08-08 03:57:21 on dist-test-slave-pww0"
server_key: "c2a870ef70b1828c8734937b7ffeddc7"
server_key_iv: "4869560f4704cc5e95f7d6b74fa7f756"
server_key_version: "encryptionkey@0"
I20260808 03:57:21.311228 31561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:21.343746 31561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:21.348237 31561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:21.348507 31561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:21.370784 31561 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:42009
I20260808 03:57:21.370806 31687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:42009 every 8 connection(s)
I20260808 03:57:21.372257 31561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:21.376449 31690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:21.379678 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 31561
I20260808 03:57:21.379937 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:21.380789 29679 external_mini_cluster.cc:1428] Setting key e8825ac55a9ba8a6ad1eb95155d4f7ed
I20260808 03:57:21.389982 31690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d: Bootstrap starting.
I20260808 03:57:21.394042 31690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:21.395385 31690 log.cc:824] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d: Log is configured to *not* fsync() on all Append() calls
Aug 08 03:57:21 dist-test-slave-pww0 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:21.398628 31690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d: No bootstrap required, opened a new log
I20260808 03:57:21.404752 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:21.383065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.28.251.254:42009)
0808 03:57:21.383910 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:21.383923 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:21.383968 (+ 45us) server_negotiation.cc:407] Connection header received
0808 03:57:21.385701 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:21.385727 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:21.386168 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:21.386556 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:21.388356 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:21.389657 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:21.391419 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:21.392187 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:21.392330 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:21.397123 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:21.397131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:21.397148 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:21.397196 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:21.400306 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:21.401064 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:21.401071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:21.401078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:21.401271 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:21.401664 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:21.401667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:21.401669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:21.402283 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:21.402577 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:21.403100 (+ 523us) server_negotiation.cc:299] Negotiation successful
0808 03:57:21.403616 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":394,"threads_started":1}
I20260808 03:57:21.407191 31690 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ad389c0e0247df9a19715375ba140d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42009 } }
I20260808 03:57:21.407610 31690 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:21.407735 31690 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ad389c0e0247df9a19715375ba140d, State: Initialized, Role: FOLLOWER
I20260808 03:57:21.408308 31690 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ad389c0e0247df9a19715375ba140d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42009 } }
I20260808 03:57:21.408551 31690 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:21.408686 31690 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:21.408856 31690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:21.411162 31690 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ad389c0e0247df9a19715375ba140d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42009 } }
I20260808 03:57:21.411751 31690 leader_election.cc:302] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5ad389c0e0247df9a19715375ba140d; no voters:
I20260808 03:57:21.412431 31690 leader_election.cc:288] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:21.412532 31698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:21.413249 31698 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [term 1 LEADER]: Becoming Leader. State: Replica: c5ad389c0e0247df9a19715375ba140d, State: Running, Role: LEADER
I20260808 03:57:21.413898 31698 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ad389c0e0247df9a19715375ba140d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42009 } }
I20260808 03:57:21.414701 31690 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:21.416716 31705 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5ad389c0e0247df9a19715375ba140d. Latest consensus state: current_term: 1 leader_uuid: "c5ad389c0e0247df9a19715375ba140d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ad389c0e0247df9a19715375ba140d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42009 } } }
I20260808 03:57:21.417104 31705 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:21.418195 31701 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5ad389c0e0247df9a19715375ba140d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ad389c0e0247df9a19715375ba140d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42009 } } }
I20260808 03:57:21.418751 31701 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:21.419889 31710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:21.425314 31710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:21.435544 31710 catalog_manager.cc:1347] Generated new cluster ID: faed43cb55fa43aab312c743afa15f5f
I20260808 03:57:21.435670 31710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:21.445896 31710 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:21.447924 31710 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:21.456670 31710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d: Generated new TSK 0
I20260808 03:57:21.457705 31710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:57:21.523051 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:42009
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:21.870218 31736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:21.870584 31736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:21.870729 31736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:21.878180 31736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:21.878312 31736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:21.878453 31736 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 03:57:21.878573 31736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:57:21.888983 31736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.251.254:42009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:21.890372 31736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:21.891835 31736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:21.905251 31746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:21.905439 31747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:21.905512 31752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:21.906303 31736 server_base.cc:1034] running on GCE node
I20260808 03:57:21.906971 31736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:21.908016 31736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:21.909277 31736 hybrid_clock.cc:648] HybridClock initialized: now 1786161441909216 us; error 51 us; skew 500 ppm
I20260808 03:57:21.909597 31736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:21 dist-test-slave-pww0 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161441, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:21.917335 31736 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:57:21.919185 31736 webserver.cc:460] Webserver started at http://127.28.251.193:35553/ using document root <none> and password file <none>
I20260808 03:57:21.920097 31736 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:21.920215 31736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:21.920559 31736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:21.923765 31736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "c3f7d66592114addb1f4f8a1431ae01f"
format_stamp: "Formatted at 2026-08-08 03:57:21 on dist-test-slave-pww0"
server_key: "eb564ca67259bcdc975303afd9d3b366"
server_key_iv: "0962b59dd752ac4cdca7aeebb17fbc32"
server_key_version: "encryptionkey@0"
I20260808 03:57:21.924600 31736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "c3f7d66592114addb1f4f8a1431ae01f"
format_stamp: "Formatted at 2026-08-08 03:57:21 on dist-test-slave-pww0"
server_key: "eb564ca67259bcdc975303afd9d3b366"
server_key_iv: "0962b59dd752ac4cdca7aeebb17fbc32"
server_key_version: "encryptionkey@0"
I20260808 03:57:21.930258 31736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 03:57:21.933993 31761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:21.935359 31736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 03:57:21.935587 31736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "c3f7d66592114addb1f4f8a1431ae01f"
format_stamp: "Formatted at 2026-08-08 03:57:21 on dist-test-slave-pww0"
server_key: "eb564ca67259bcdc975303afd9d3b366"
server_key_iv: "0962b59dd752ac4cdca7aeebb17fbc32"
server_key_version: "encryptionkey@0"
I20260808 03:57:21.935832 31736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:21.960013 31736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:21.964656 31736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:21.964972 31736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:21.968003 31736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:21.968101 31736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:21.968233 31736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:21.968302 31736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 03:57:21 dist-test-slave-pww0 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161441, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:21.993701 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:21.968668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:35105 (local address 127.28.251.254:42009)
0808 03:57:21.969204 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:21.969211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:21.970458 (+ 1247us) server_negotiation.cc:407] Connection header received
0808 03:57:21.972801 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:21.972809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:21.972910 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:21.973047 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:21.975381 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:21.976428 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:21.979498 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:21.980013 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:21.980166 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:21.985605 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:21.985614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:21.985619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:21.985673 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:21.989360 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:21.990581 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:21.990588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:21.990591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:21.990788 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:21.991402 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:21.991406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:21.991409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:21.991845 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:21.992021 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:21.993157 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0808 03:57:21.993361 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":254,"threads_started":1}
I20260808 03:57:21.995393 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:21.969262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:42009 (local address 127.28.251.193:35105)
0808 03:57:21.970214 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:21.970272 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:21.972455 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:21.973345 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:21.973355 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:21.974177 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:21.975147 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:21.975177 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:21.976615 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:21.976621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:21.979261 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:21.979284 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:21.980309 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:21.980319 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:21.980690 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:21.981652 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:21.981703 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:21.985227 (+ 3524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:21.989580 (+ 4353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:21.989593 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:21.989626 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:21.990416 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:21.990949 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:21.990953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:21.990955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:21.991258 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:21.992049 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:21.992060 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:21.992634 (+ 574us) client_negotiation.cc:769] Sending connection context
0808 03:57:21.993008 (+ 374us) client_negotiation.cc:240] Negotiation successful
0808 03:57:21.993682 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":376,"threads_started":1}
I20260808 03:57:22.013434 31736 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:33963
I20260808 03:57:22.013473 31954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:33963 every 8 connection(s)
I20260808 03:57:22.014909 31736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:57:22.017362 31956 heartbeater.cc:347] Connected to a master server at 127.28.251.254:42009
I20260808 03:57:22.017586 31956 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:22.018291 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 31736
I20260808 03:57:22.018519 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:57:22.018520 31956 heartbeater.cc:511] Master 127.28.251.254:42009 requested a full tablet report, sending...
I20260808 03:57:22.018836 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:22.003419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:43187 (local address 127.28.251.254:42009)
0808 03:57:22.003655 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:22.003661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:22.004424 (+ 763us) server_negotiation.cc:407] Connection header received
0808 03:57:22.004611 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:22.004617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:22.004718 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:22.004847 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:22.005946 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.007086 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.008976 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.009682 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.009823 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:22.012423 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:22.012430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:22.012435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:22.012480 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:22.015711 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:22.016705 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:22.016711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:22.016714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:22.016878 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:22.017279 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:22.017285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:22.017288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:22.017642 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:22.017815 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:22.018254 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 03:57:22.018480 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260808 03:57:22.018965 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:22.003502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:42009 (local address 127.28.251.193:43187)
0808 03:57:22.004253 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:22.004272 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:22.004455 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:22.004924 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:22.004933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:22.005420 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:22.005770 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:22.005782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.007228 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.007233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:22.008792 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:22.008814 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.009939 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.009945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:22.010129 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:22.010969 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:22.010996 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:22.012173 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:22.015893 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:22.015899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:22.015903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:22.016556 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:22.017010 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:22.017014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:22.017016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:22.017153 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:22.017761 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:22.017766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:22.017967 (+ 201us) client_negotiation.cc:769] Sending connection context
0808 03:57:22.018094 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 03:57:22.018473 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":397,"threads_started":1}
I20260808 03:57:22.019361 29679 external_mini_cluster.cc:1428] Setting key c17c668c587396f6bd792985f3f9994c
I20260808 03:57:22.021490 31617 ts_manager.cc:194] Registered new tserver with Master: c3f7d66592114addb1f4f8a1431ae01f (127.28.251.193:33963)
I20260808 03:57:22.023516 31617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:35105
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:57:22.072027 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:42009
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:22.431736 31962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:22.432070 31962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:22.432176 31962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:22.440234 31962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:22.440378 31962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:22.440501 31962 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 03:57:22.440603 31962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:57:22.451843 31962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.251.254:42009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:22.453349 31962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:22.454844 31962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:22.469880 31976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:22.469971 31979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:22.469904 31975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:22.470608 31962 server_base.cc:1034] running on GCE node
I20260808 03:57:22.471319 31962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:22.472432 31962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:22.473685 31962 hybrid_clock.cc:648] HybridClock initialized: now 1786161442473590 us; error 58 us; skew 500 ppm
I20260808 03:57:22.474020 31962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:22 dist-test-slave-pww0 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161442, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:22.477548 31962 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:57:22.479200 31962 webserver.cc:460] Webserver started at http://127.28.251.194:43413/ using document root <none> and password file <none>
I20260808 03:57:22.480043 31962 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:22.480144 31962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:22.480477 31962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:22.483758 31962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "ae10b034db5b43fc805fb1285d5e991c"
format_stamp: "Formatted at 2026-08-08 03:57:22 on dist-test-slave-pww0"
server_key: "2e6b0801e7c07e5f34bc12803b3a1fb0"
server_key_iv: "c3c942b0b861246812622fc5bfe248e1"
server_key_version: "encryptionkey@0"
I20260808 03:57:22.484624 31962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "ae10b034db5b43fc805fb1285d5e991c"
format_stamp: "Formatted at 2026-08-08 03:57:22 on dist-test-slave-pww0"
server_key: "2e6b0801e7c07e5f34bc12803b3a1fb0"
server_key_iv: "c3c942b0b861246812622fc5bfe248e1"
server_key_version: "encryptionkey@0"
I20260808 03:57:22.490409 31962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 03:57:22.494239 31989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:22.495883 31962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:22.496070 31962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "ae10b034db5b43fc805fb1285d5e991c"
format_stamp: "Formatted at 2026-08-08 03:57:22 on dist-test-slave-pww0"
server_key: "2e6b0801e7c07e5f34bc12803b3a1fb0"
server_key_iv: "c3c942b0b861246812622fc5bfe248e1"
server_key_version: "encryptionkey@0"
I20260808 03:57:22.496300 31962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:22.509164 31962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:22.513863 31962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:22.514155 31962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:22.517400 31962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:22.517491 31962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:22.517628 31962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:22.517704 31962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 03:57:22 dist-test-slave-pww0 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161442, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:22.543373 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:22.518172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:57679 (local address 127.28.251.254:42009)
0808 03:57:22.518361 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:22.518367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:22.520016 (+ 1649us) server_negotiation.cc:407] Connection header received
0808 03:57:22.522506 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:22.522511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:22.522599 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:22.522685 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:22.524830 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.525889 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.529379 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.529973 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.530123 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:22.535480 (+ 5357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:22.535490 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:22.535496 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:22.535559 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:22.539233 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:22.540631 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:22.540636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:22.540639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:22.540812 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:22.541355 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:22.541361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:22.541364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:22.541679 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:22.541848 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:22.542727 (+ 879us) server_negotiation.cc:299] Negotiation successful
0808 03:57:22.542998 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 03:57:22.544576 32001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:22.518771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:42009 (local address 127.28.251.194:57679)
0808 03:57:22.519798 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:22.519848 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:22.522096 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:22.523303 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:22.523316 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:22.523968 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:22.524611 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:22.524630 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.526119 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.526126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:22.529109 (+ 2983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:22.529140 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.530264 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.530274 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:22.530564 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:22.531397 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:22.531423 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:22.535042 (+ 3619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:22.539497 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:22.539510 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:22.539538 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:22.540348 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:22.540946 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:22.540950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:22.540953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:22.541199 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:22.541846 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:22.541857 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:22.542279 (+ 422us) client_negotiation.cc:769] Sending connection context
0808 03:57:22.542558 (+ 279us) client_negotiation.cc:240] Negotiation successful
0808 03:57:22.543217 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":424,"threads_started":1}
I20260808 03:57:22.567085 31962 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:37739
I20260808 03:57:22.567106 32173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:37739 every 8 connection(s)
I20260808 03:57:22.567667 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:22.552653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:42009 (local address 127.28.251.194:36733)
0808 03:57:22.553430 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:22.553452 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:22.553714 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:22.554114 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:22.554119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:22.554803 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:22.555270 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:22.555284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.556412 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.556416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:22.557606 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:22.557622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.558475 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.558483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:22.558645 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:22.559296 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:22.559316 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:22.560484 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:22.564365 (+ 3881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:22.564372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:22.564376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:22.565024 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:22.565488 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:22.565493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:22.565497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:22.565647 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:22.566409 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:22.566414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:22.566660 (+ 246us) client_negotiation.cc:769] Sending connection context
0808 03:57:22.566842 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 03:57:22.567196 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":403,"threads_started":1}
I20260808 03:57:22.568152 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:22.553215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:36733 (local address 127.28.251.254:42009)
0808 03:57:22.553430 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:22.553435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:22.553564 (+ 129us) server_negotiation.cc:407] Connection header received
0808 03:57:22.553875 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:22.553881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:22.553958 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:22.554045 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:22.555488 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.556232 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.557784 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:22.558244 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:22.558378 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:22.560752 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:22.560760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:22.560764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:22.560816 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:22.564152 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:22.565192 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:22.565195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:22.565197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:22.565355 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:22.565779 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:22.565783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:22.565785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:22.566274 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:22.566476 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:22.567658 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0808 03:57:22.567927 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 03:57:22.568908 31962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:57:22.572067 32175 heartbeater.cc:347] Connected to a master server at 127.28.251.254:42009
I20260808 03:57:22.572335 32175 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:22.573096 32175 heartbeater.cc:511] Master 127.28.251.254:42009 requested a full tablet report, sending...
I20260808 03:57:22.574522 31617 ts_manager.cc:194] Registered new tserver with Master: ae10b034db5b43fc805fb1285d5e991c (127.28.251.194:37739)
I20260808 03:57:22.575387 31617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:57679
I20260808 03:57:22.577523 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 31962
I20260808 03:57:22.577759 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:57:22.578322 29679 external_mini_cluster.cc:1428] Setting key 0441222bcdea54751e9638aa1110359a
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:57:22.631668 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:42009
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:22.986055 32181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:22.986373 32181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:22.986507 32181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:22.994004 32181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:22.994124 32181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:22.994239 32181 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 03:57:22.994347 32181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:57:23.005116 32181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.251.254:42009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:23.006600 32181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:23.008245 32181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:23.023432 32188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:23.023487 32187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:23.024019 32191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:23.024101 32181 server_base.cc:1034] running on GCE node
I20260808 03:57:23.024787 32181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:23.025810 32181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:23.026818 31956 heartbeater.cc:503] Master 127.28.251.254:42009 was elected leader, sending a full tablet report...
I20260808 03:57:23.027081 32181 hybrid_clock.cc:648] HybridClock initialized: now 1786161443026998 us; error 54 us; skew 500 ppm
I20260808 03:57:23.027554 32181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:23 dist-test-slave-pww0 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161443, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:23.031124 32181 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:57:23.032765 32181 webserver.cc:460] Webserver started at http://127.28.251.195:45081/ using document root <none> and password file <none>
I20260808 03:57:23.033509 32181 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:23.033608 32181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:23.033893 32181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:23.037197 32181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "5bc8581e2a3e4e769d66bb45750f0b48"
format_stamp: "Formatted at 2026-08-08 03:57:23 on dist-test-slave-pww0"
server_key: "cc96a55e6d7de1e800d5749393fad3db"
server_key_iv: "d191b5d76705ee9b333ff235ac12954b"
server_key_version: "encryptionkey@0"
I20260808 03:57:23.038055 32181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "5bc8581e2a3e4e769d66bb45750f0b48"
format_stamp: "Formatted at 2026-08-08 03:57:23 on dist-test-slave-pww0"
server_key: "cc96a55e6d7de1e800d5749393fad3db"
server_key_iv: "d191b5d76705ee9b333ff235ac12954b"
server_key_version: "encryptionkey@0"
I20260808 03:57:23.043880 32181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 03:57:23.047771 32201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:23.049238 32181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:57:23.049409 32181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "5bc8581e2a3e4e769d66bb45750f0b48"
format_stamp: "Formatted at 2026-08-08 03:57:23 on dist-test-slave-pww0"
server_key: "cc96a55e6d7de1e800d5749393fad3db"
server_key_iv: "d191b5d76705ee9b333ff235ac12954b"
server_key_version: "encryptionkey@0"
I20260808 03:57:23.049629 32181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:23.070273 32181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:23.074985 32181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:23.075286 32181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:23.078214 32181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:23.078311 32181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:23.078465 32181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:23.078526 32181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 03:57:23 dist-test-slave-pww0 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161443, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:23.103991 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.078928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:39493 (local address 127.28.251.254:42009)
0808 03:57:23.079511 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.079517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.080737 (+ 1220us) server_negotiation.cc:407] Connection header received
0808 03:57:23.082911 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.082917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.083019 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.083152 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:23.085739 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.086972 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.089453 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.090011 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.090151 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.096018 (+ 5867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:23.096026 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:23.096032 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:23.096085 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:23.099621 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:23.100907 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:23.100914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:23.100917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:23.101097 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:23.101764 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:23.101770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:23.101773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:23.102077 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:23.102230 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.103237 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.103554 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":304,"threads_started":1}
I20260808 03:57:23.105031 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.079610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:42009 (local address 127.28.251.195:39493)
0808 03:57:23.080534 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:23.080584 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:23.082537 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:23.083588 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:23.083600 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:23.084524 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:23.085486 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.085512 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.087188 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.087194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.089224 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.089262 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.090551 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.090559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.090906 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.091878 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:23.091922 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:23.095709 (+ 3787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:23.099825 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:23.099840 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:23.099866 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:23.100722 (+ 856us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:23.101250 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:23.101254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:23.101264 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:23.101586 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:23.102230 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:23.102243 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:23.102671 (+ 428us) client_negotiation.cc:769] Sending connection context
0808 03:57:23.103065 (+ 394us) client_negotiation.cc:240] Negotiation successful
0808 03:57:23.103605 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":382,"threads_started":1}
I20260808 03:57:23.128676 32344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.112406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:42009 (local address 127.28.251.195:44655)
0808 03:57:23.113104 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:23.113133 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:23.113310 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:23.113899 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:23.113904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:23.114521 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:23.114932 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.114948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.116638 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.116644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.118449 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.118466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.119528 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.119536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.119753 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.120516 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:23.120544 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:23.121719 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:23.125250 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:23.125256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:23.125261 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:23.125877 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:23.126344 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:23.126350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:23.126354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:23.126536 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:23.127245 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:23.127250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:23.127523 (+ 273us) client_negotiation.cc:769] Sending connection context
0808 03:57:23.127702 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 03:57:23.128117 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":371,"threads_started":1}
I20260808 03:57:23.128854 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.112506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:44655 (local address 127.28.251.254:42009)
0808 03:57:23.112733 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.112740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.113228 (+ 488us) server_negotiation.cc:407] Connection header received
0808 03:57:23.113510 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.113514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.113583 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.113641 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:23.115096 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.116371 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.118676 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.119250 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.119400 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.121949 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:23.121954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:23.121957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:23.121993 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:23.125092 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:23.126024 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:23.126030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:23.126034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:23.126204 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:23.126682 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:23.126687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:23.126690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:23.127116 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:23.127281 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.127963 (+ 682us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.128334 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 03:57:23.143541 32181 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:43467
I20260808 03:57:23.143577 32391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:43467 every 8 connection(s)
I20260808 03:57:23.145437 32181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:57:23.146503 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 32181
I20260808 03:57:23.146718 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:57:23.147289 29679 external_mini_cluster.cc:1428] Setting key e6bc8f744757cbc22aff5eb9b9d0f9f1
I20260808 03:57:23.148095 32392 heartbeater.cc:347] Connected to a master server at 127.28.251.254:42009
I20260808 03:57:23.148375 32392 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:23.149348 32392 heartbeater.cc:511] Master 127.28.251.254:42009 requested a full tablet report, sending...
I20260808 03:57:23.152662 31617 ts_manager.cc:194] Registered new tserver with Master: 5bc8581e2a3e4e769d66bb45750f0b48 (127.28.251.195:43467)
I20260808 03:57:23.154045 31617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:39493
I20260808 03:57:23.164247 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 03:57:23 dist-test-slave-pww0 krb5kdc[31545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 03:57:23 dist-test-slave-pww0 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:23.220556 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.205940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.28.251.254:42009)
0808 03:57:23.206169 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.206174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.206515 (+ 341us) server_negotiation.cc:407] Connection header received
0808 03:57:23.206648 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.206662 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.206800 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.206907 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:23.208613 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.209367 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.210674 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.211105 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.211201 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.215254 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:23.215260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:23.215263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:23.215309 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:23.218061 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:23.218734 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:23.218737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:23.218740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:23.218877 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:23.219187 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:23.219190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:23.219192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:23.219534 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:23.219745 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.219998 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.220216 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 03:57:23.224884 31617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:57:23.231956 31617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:57:23.256860 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.248097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.28.251.195:43467)
0808 03:57:23.249667 (+ 1570us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.249673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.249698 (+ 25us) server_negotiation.cc:407] Connection header received
0808 03:57:23.249781 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.249786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.249939 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.250091 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:23.250719 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.252108 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.254336 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.255444 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.255663 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.255868 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.256091 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.256269 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1400,"thread_start_us":1222,"threads_started":1}
I20260808 03:57:23.258016 32412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.246205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33963 (local address 127.0.0.1:51860)
0808 03:57:23.247634 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:23.247653 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:23.247808 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:23.248990 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:23.248994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:23.249019 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:23.249441 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.249454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.251359 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.251364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.254279 (+ 2915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.254298 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.256310 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.256318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.256491 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.256522 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 03:57:23.256602 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 03:57:23.257543 (+ 941us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"mutex_wait_us":291,"thread_start_us":1067,"threads_started":1}
I20260808 03:57:23.258417 32410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.245033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:37739 (local address 127.0.0.1:55426)
0808 03:57:23.245856 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:23.245916 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:23.246109 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:23.247549 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:23.247557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:23.247604 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:23.248097 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.248118 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.251420 (+ 3302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.251424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.254348 (+ 2924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.254359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.257526 (+ 3167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.257532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.257724 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.257758 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 03:57:23.257849 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 03:57:23.257983 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":333,"thread_start_us":381,"threads_started":1}
I20260808 03:57:23.258622 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.245425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.28.251.194:37739)
0808 03:57:23.246803 (+ 1378us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.246814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.246845 (+ 31us) server_negotiation.cc:407] Connection header received
0808 03:57:23.246958 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.246966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.247272 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.247546 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:23.249621 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.251213 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.254521 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.255959 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.256331 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.256530 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.258127 (+ 1597us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.258321 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":1093,"threads_started":1}
I20260808 03:57:23.259038 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.247897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:43467 (local address 127.0.0.1:55866)
0808 03:57:23.248461 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:23.248480 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:23.248596 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:23.250111 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:23.250115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:23.250138 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:23.250510 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.250521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.252279 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.252284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.254144 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.254161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.255645 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.255651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.255844 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.255897 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 03:57:23.256023 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 03:57:23.257546 (+ 1523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":7,"thread_start_us":297,"threads_started":1}
I20260808 03:57:23.259670 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.246498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.28.251.193:33963)
0808 03:57:23.247130 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.247137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.248428 (+ 1291us) server_negotiation.cc:407] Connection header received
0808 03:57:23.248545 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.248553 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.248800 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.248984 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:23.249609 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.251207 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.254511 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.255976 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.257854 (+ 1878us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.258033 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.258223 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.258381 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":328,"threads_started":1}
I20260808 03:57:23.262830 31846 tablet_service.cc:1467] Processing CreateTablet for tablet 974390227b064c05bc7eb58d005eb405 (DEFAULT_TABLE table=test-table [id=b9f383882cd047a79119b4ce628bb73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:23.262768 32065 tablet_service.cc:1467] Processing CreateTablet for tablet 974390227b064c05bc7eb58d005eb405 (DEFAULT_TABLE table=test-table [id=b9f383882cd047a79119b4ce628bb73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:23.262768 32277 tablet_service.cc:1467] Processing CreateTablet for tablet 974390227b064c05bc7eb58d005eb405 (DEFAULT_TABLE table=test-table [id=b9f383882cd047a79119b4ce628bb73b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:23.264228 31846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974390227b064c05bc7eb58d005eb405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:23.264303 32277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974390227b064c05bc7eb58d005eb405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:23.264305 32065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974390227b064c05bc7eb58d005eb405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:23.277954 32418 tablet_bootstrap.cc:492] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: Bootstrap starting.
I20260808 03:57:23.280069 32421 tablet_bootstrap.cc:492] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: Bootstrap starting.
I20260808 03:57:23.281716 32419 tablet_bootstrap.cc:492] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: Bootstrap starting.
I20260808 03:57:23.282651 32418 tablet_bootstrap.cc:654] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:23.284116 32421 tablet_bootstrap.cc:654] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:23.284165 32418 log.cc:824] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:23.284873 32419 tablet_bootstrap.cc:654] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:23.285457 32421 log.cc:824] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:23.286134 32419 log.cc:824] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:23.288496 32418 tablet_bootstrap.cc:492] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: No bootstrap required, opened a new log
I20260808 03:57:23.288813 32418 ts_tablet_manager.cc:1397] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20260808 03:57:23.289098 32419 tablet_bootstrap.cc:492] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: No bootstrap required, opened a new log
I20260808 03:57:23.289345 32419 ts_tablet_manager.cc:1397] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 03:57:23.290156 32421 tablet_bootstrap.cc:492] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: No bootstrap required, opened a new log
I20260808 03:57:23.290470 32421 ts_tablet_manager.cc:1397] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260808 03:57:23.296892 32419 raft_consensus.cc:348] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.297762 32419 raft_consensus.cc:374] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:23.297892 32419 raft_consensus.cc:729] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc8581e2a3e4e769d66bb45750f0b48, State: Initialized, Role: FOLLOWER
I20260808 03:57:23.298676 32419 consensus_queue.cc:260] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.299929 32392 heartbeater.cc:503] Master 127.28.251.254:42009 was elected leader, sending a full tablet report...
I20260808 03:57:23.300189 32418 raft_consensus.cc:348] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.300680 32418 raft_consensus.cc:374] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:23.300788 32418 raft_consensus.cc:729] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f7d66592114addb1f4f8a1431ae01f, State: Initialized, Role: FOLLOWER
I20260808 03:57:23.301427 32418 consensus_queue.cc:260] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.301842 32421 raft_consensus.cc:348] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.302320 32421 raft_consensus.cc:374] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:23.302425 32421 raft_consensus.cc:729] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae10b034db5b43fc805fb1285d5e991c, State: Initialized, Role: FOLLOWER
I20260808 03:57:23.302446 32419 ts_tablet_manager.cc:1428] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260808 03:57:23.303048 32421 consensus_queue.cc:260] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.308779 32175 heartbeater.cc:503] Master 127.28.251.254:42009 was elected leader, sending a full tablet report...
I20260808 03:57:23.309445 32421 ts_tablet_manager.cc:1428] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: Time spent starting tablet: real 0.019s user 0.015s sys 0.005s
I20260808 03:57:23.310325 32418 ts_tablet_manager.cc:1428] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: Time spent starting tablet: real 0.021s user 0.008s sys 0.013s
W20260808 03:57:23.321274 32177 tablet.cc:2369] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 03:57:23.397475 32393 tablet.cc:2369] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:23.501637 32431 raft_consensus.cc:482] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:23.501999 32431 raft_consensus.cc:504] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.504062 32431 leader_election.cc:288] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3f7d66592114addb1f4f8a1431ae01f (127.28.251.193:33963), 5bc8581e2a3e4e769d66bb45750f0b48 (127.28.251.195:43467)
I20260808 03:57:23.511111 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.504603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:50599 (local address 127.28.251.193:33963)
0808 03:57:23.504886 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.504893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.505311 (+ 418us) server_negotiation.cc:407] Connection header received
0808 03:57:23.505540 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.505547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.505655 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.505756 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:23.506422 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.507265 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.509342 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.510127 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.510289 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.510395 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.510720 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.510870 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 03:57:23.511387 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.505656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:55047 (local address 127.28.251.195:43467)
0808 03:57:23.505868 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.505874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.506096 (+ 222us) server_negotiation.cc:407] Connection header received
0808 03:57:23.506283 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.506286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.506346 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.506421 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:23.507087 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.507907 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.509660 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.510470 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.510625 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.510700 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.511009 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.511137 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 03:57:23.511606 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.504499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33963 (local address 127.28.251.194:50599)
0808 03:57:23.505173 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:23.505190 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:23.505364 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:23.505883 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:23.505887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:23.505918 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:23.506263 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.506273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.507413 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.507417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.509160 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.509177 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.510400 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.510409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.510552 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.510578 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 03:57:23.510672 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 03:57:23.510813 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":52,"thread_start_us":367,"threads_started":1}
I20260808 03:57:23.511629 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.505318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:43467 (local address 127.28.251.194:55047)
0808 03:57:23.505989 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:23.506003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:23.506136 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:23.506463 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:23.506468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:23.506493 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:23.506890 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.506901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.508048 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.508053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.509491 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.509504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.510714 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.510722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.510855 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.510879 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 03:57:23.510961 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 03:57:23.511092 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":2,"thread_start_us":290,"threads_started":1}
I20260808 03:57:23.512682 32314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974390227b064c05bc7eb58d005eb405" candidate_uuid: "ae10b034db5b43fc805fb1285d5e991c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" is_pre_election: true
I20260808 03:57:23.512681 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974390227b064c05bc7eb58d005eb405" candidate_uuid: "ae10b034db5b43fc805fb1285d5e991c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f7d66592114addb1f4f8a1431ae01f" is_pre_election: true
I20260808 03:57:23.513185 31877 raft_consensus.cc:2454] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae10b034db5b43fc805fb1285d5e991c in term 0.
I20260808 03:57:23.513185 32314 raft_consensus.cc:2454] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae10b034db5b43fc805fb1285d5e991c in term 0.
I20260808 03:57:23.513852 31991 leader_election.cc:302] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae10b034db5b43fc805fb1285d5e991c, c3f7d66592114addb1f4f8a1431ae01f; no voters:
I20260808 03:57:23.514271 32431 raft_consensus.cc:2781] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:57:23.514458 32431 raft_consensus.cc:482] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:57:23.514575 32431 raft_consensus.cc:3037] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:23.517215 32431 raft_consensus.cc:504] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.517860 32431 leader_election.cc:288] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [CANDIDATE]: Term 1 election: Requested vote from peers c3f7d66592114addb1f4f8a1431ae01f (127.28.251.193:33963), 5bc8581e2a3e4e769d66bb45750f0b48 (127.28.251.195:43467)
W20260808 03:57:23.518101 31959 tablet.cc:2369] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:23.518563 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974390227b064c05bc7eb58d005eb405" candidate_uuid: "ae10b034db5b43fc805fb1285d5e991c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f7d66592114addb1f4f8a1431ae01f"
I20260808 03:57:23.518615 32314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974390227b064c05bc7eb58d005eb405" candidate_uuid: "ae10b034db5b43fc805fb1285d5e991c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc8581e2a3e4e769d66bb45750f0b48"
I20260808 03:57:23.518806 31877 raft_consensus.cc:3037] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:23.518839 32314 raft_consensus.cc:3037] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:23.521303 32314 raft_consensus.cc:2454] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae10b034db5b43fc805fb1285d5e991c in term 1.
I20260808 03:57:23.521450 31877 raft_consensus.cc:2454] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae10b034db5b43fc805fb1285d5e991c in term 1.
I20260808 03:57:23.521808 31991 leader_election.cc:302] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bc8581e2a3e4e769d66bb45750f0b48, ae10b034db5b43fc805fb1285d5e991c; no voters:
I20260808 03:57:23.522133 32431 raft_consensus.cc:2781] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:23.522622 32431 raft_consensus.cc:686] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 1 LEADER]: Becoming Leader. State: Replica: ae10b034db5b43fc805fb1285d5e991c, State: Running, Role: LEADER
I20260808 03:57:23.523175 32431 consensus_queue.cc:237] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.528723 31618 catalog_manager.cc:5495] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c reported cstate change: term changed from 0 to 1, leader changed from <none> to ae10b034db5b43fc805fb1285d5e991c (127.28.251.194). New cstate: current_term: 1 leader_uuid: "ae10b034db5b43fc805fb1285d5e991c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } health_report { overall_health: UNKNOWN } } }
I20260808 03:57:23.563104 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.557296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.28.251.194:37739)
0808 03:57:23.557520 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.557527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.557658 (+ 131us) server_negotiation.cc:407] Connection header received
0808 03:57:23.558019 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.558024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.558110 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.558242 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:23.558667 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.559416 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.560817 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.561228 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.561342 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.561909 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:23.562437 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:23.562575 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.562735 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.562827 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 03:57:23.572728 31877 raft_consensus.cc:1261] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f [term 1 FOLLOWER]: Refusing update from remote peer ae10b034db5b43fc805fb1285d5e991c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:23.572738 32314 raft_consensus.cc:1261] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 1 FOLLOWER]: Refusing update from remote peer ae10b034db5b43fc805fb1285d5e991c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:23.573954 32434 consensus_queue.cc:1035] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:23.574361 32431 consensus_queue.cc:1035] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:23.607486 31618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:23.607788 31618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:23.612370 31618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d: Sending DeleteTablet for 3 replicas of tablet 974390227b064c05bc7eb58d005eb405
I20260808 03:57:23.613755 32065 tablet_service.cc:1514] Processing DeleteTablet for tablet 974390227b064c05bc7eb58d005eb405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:23 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:55426
I20260808 03:57:23.613839 32277 tablet_service.cc:1514] Processing DeleteTablet for tablet 974390227b064c05bc7eb58d005eb405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:23 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:55866
I20260808 03:57:23.614651 31846 tablet_service.cc:1514] Processing DeleteTablet for tablet 974390227b064c05bc7eb58d005eb405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:23 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:51860
I20260808 03:57:23.616101 31618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:57:23.616909 31618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:57:23.619817 32469 tablet_replica.cc:330] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: stopping tablet replica
I20260808 03:57:23.620443 32469 raft_consensus.cc:2229] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:23.620985 32469 raft_consensus.cc:2258] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:23.625458 32470 tablet_replica.cc:330] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: stopping tablet replica
I20260808 03:57:23.626010 32470 raft_consensus.cc:2229] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:23.626591 32470 raft_consensus.cc:2258] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:23.627502 32466 tablet_replica.cc:330] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: stopping tablet replica
I20260808 03:57:23.627507 31845 tablet_service.cc:1467] Processing CreateTablet for tablet 7b22fd43994b441aa183f2531f118ea7 (DEFAULT_TABLE table=test-table [id=5aa666bea02d41be9c3af5edb4499fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:23.628026 31845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b22fd43994b441aa183f2531f118ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:23.628005 32466 raft_consensus.cc:2229] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:57:23.628702 32466 raft_consensus.cc:2258] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:23.629009 32470 ts_tablet_manager.cc:1905] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:23.629825 32062 tablet_service.cc:1467] Processing CreateTablet for tablet 7b22fd43994b441aa183f2531f118ea7 (DEFAULT_TABLE table=test-table [id=5aa666bea02d41be9c3af5edb4499fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:23.630424 32062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b22fd43994b441aa183f2531f118ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:23.631258 32466 ts_tablet_manager.cc:1905] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:23.635087 32418 tablet_bootstrap.cc:492] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f: Bootstrap starting.
I20260808 03:57:23.636821 32418 tablet_bootstrap.cc:654] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:23.637101 32469 ts_tablet_manager.cc:1905] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:23.638271 32277 tablet_service.cc:1467] Processing CreateTablet for tablet 7b22fd43994b441aa183f2531f118ea7 (DEFAULT_TABLE table=test-table [id=5aa666bea02d41be9c3af5edb4499fb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:23.638800 32277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b22fd43994b441aa183f2531f118ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:23.640537 32421 tablet_bootstrap.cc:492] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c: Bootstrap starting.
I20260808 03:57:23.641539 32466 ts_tablet_manager.cc:1918] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:57:23.641733 32466 log.cc:1196] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/974390227b064c05bc7eb58d005eb405
I20260808 03:57:23.642261 32466 ts_tablet_manager.cc:1939] T 974390227b064c05bc7eb58d005eb405 P ae10b034db5b43fc805fb1285d5e991c: Deleting consensus metadata
I20260808 03:57:23.642433 32421 tablet_bootstrap.cc:654] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:23.644080 32469 ts_tablet_manager.cc:1918] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:57:23.644256 32469 log.cc:1196] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/974390227b064c05bc7eb58d005eb405
I20260808 03:57:23.644675 31584 catalog_manager.cc:4841] TS ae10b034db5b43fc805fb1285d5e991c (127.28.251.194:37739): tablet 974390227b064c05bc7eb58d005eb405 (table test-table [id=b9f383882cd047a79119b4ce628bb73b]) successfully deleted
I20260808 03:57:23.644781 32469 ts_tablet_manager.cc:1939] T 974390227b064c05bc7eb58d005eb405 P 5bc8581e2a3e4e769d66bb45750f0b48: Deleting consensus metadata
I20260808 03:57:23.646879 31584 catalog_manager.cc:4841] TS 5bc8581e2a3e4e769d66bb45750f0b48 (127.28.251.195:43467): tablet 974390227b064c05bc7eb58d005eb405 (table test-table [id=b9f383882cd047a79119b4ce628bb73b]) successfully deleted
I20260808 03:57:23.647922 32419 tablet_bootstrap.cc:492] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48: Bootstrap starting.
I20260808 03:57:23.648972 32470 ts_tablet_manager.cc:1918] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:57:23.649165 32470 log.cc:1196] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/974390227b064c05bc7eb58d005eb405
I20260808 03:57:23.649658 32419 tablet_bootstrap.cc:654] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:23.649736 32470 ts_tablet_manager.cc:1939] T 974390227b064c05bc7eb58d005eb405 P c3f7d66592114addb1f4f8a1431ae01f: Deleting consensus metadata
I20260808 03:57:23.650027 32418 tablet_bootstrap.cc:492] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f: No bootstrap required, opened a new log
I20260808 03:57:23.650185 32418 ts_tablet_manager.cc:1397] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260808 03:57:23.651124 32418 raft_consensus.cc:348] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.651373 32418 raft_consensus.cc:374] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:23.651503 32418 raft_consensus.cc:729] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f7d66592114addb1f4f8a1431ae01f, State: Initialized, Role: FOLLOWER
I20260808 03:57:23.651839 32418 consensus_queue.cc:260] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.652455 32418 ts_tablet_manager.cc:1428] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 03:57:23.654294 32421 tablet_bootstrap.cc:492] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c: No bootstrap required, opened a new log
I20260808 03:57:23.654484 32421 ts_tablet_manager.cc:1397] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20260808 03:57:23.654466 31584 catalog_manager.cc:4841] TS c3f7d66592114addb1f4f8a1431ae01f (127.28.251.193:33963): tablet 974390227b064c05bc7eb58d005eb405 (table test-table [id=b9f383882cd047a79119b4ce628bb73b]) successfully deleted
I20260808 03:57:23.655548 32421 raft_consensus.cc:348] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.655818 32421 raft_consensus.cc:374] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:23.655937 32421 raft_consensus.cc:729] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae10b034db5b43fc805fb1285d5e991c, State: Initialized, Role: FOLLOWER
I20260808 03:57:23.656229 32421 consensus_queue.cc:260] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.656895 32421 ts_tablet_manager.cc:1428] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 03:57:23.657688 32419 tablet_bootstrap.cc:492] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48: No bootstrap required, opened a new log
I20260808 03:57:23.657859 32419 ts_tablet_manager.cc:1397] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260808 03:57:23.658895 32419 raft_consensus.cc:348] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.659135 32419 raft_consensus.cc:374] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:23.659243 32419 raft_consensus.cc:729] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc8581e2a3e4e769d66bb45750f0b48, State: Initialized, Role: FOLLOWER
I20260808 03:57:23.659564 32419 consensus_queue.cc:260] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.660753 32419 ts_tablet_manager.cc:1428] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 03:57:23.807400 32428 raft_consensus.cc:482] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:23.807742 32428 raft_consensus.cc:504] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.809837 32428 leader_election.cc:288] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae10b034db5b43fc805fb1285d5e991c (127.28.251.194:37739), 5bc8581e2a3e4e769d66bb45750f0b48 (127.28.251.195:43467)
I20260808 03:57:23.818301 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.810695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:60155 (local address 127.28.251.194:37739)
0808 03:57:23.810922 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.810929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.811061 (+ 132us) server_negotiation.cc:407] Connection header received
0808 03:57:23.811351 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.811358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.811446 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.811549 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:23.812458 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.813492 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.816254 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.817292 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.817448 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.817533 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.817864 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.817997 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 03:57:23.818609 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.810296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:37739 (local address 127.28.251.193:60155)
0808 03:57:23.810944 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:23.810968 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:23.811150 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:23.811811 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:23.811817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:23.811850 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:23.812254 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.812265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.813633 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.813639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.816066 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.816086 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.817518 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.817524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.817690 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.817716 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 03:57:23.817801 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 03:57:23.817938 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":26,"thread_start_us":479,"threads_started":1}
I20260808 03:57:23.819100 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.811448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:32841 (local address 127.28.251.195:43467)
0808 03:57:23.811663 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.811668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.812392 (+ 724us) server_negotiation.cc:407] Connection header received
0808 03:57:23.812550 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.812555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.812637 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.812728 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:23.813417 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.814234 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.816628 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.817855 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.818007 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.818094 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.818694 (+ 600us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.818840 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 03:57:23.819203 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.811258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:43467 (local address 127.28.251.193:32841)
0808 03:57:23.812250 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:23.812269 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:23.812413 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:23.812779 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:23.812784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:23.812811 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:23.813244 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.813257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.814379 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.814383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.816475 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.816491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.818279 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.818285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.818468 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.818502 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 03:57:23.818645 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 03:57:23.818786 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":82,"thread_start_us":568,"threads_started":1}
I20260808 03:57:23.819941 32093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b22fd43994b441aa183f2531f118ea7" candidate_uuid: "c3f7d66592114addb1f4f8a1431ae01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae10b034db5b43fc805fb1285d5e991c" is_pre_election: true
I20260808 03:57:23.820317 32314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b22fd43994b441aa183f2531f118ea7" candidate_uuid: "c3f7d66592114addb1f4f8a1431ae01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" is_pre_election: true
I20260808 03:57:23.820482 32093 raft_consensus.cc:2454] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f7d66592114addb1f4f8a1431ae01f in term 0.
I20260808 03:57:23.820562 32314 raft_consensus.cc:2454] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3f7d66592114addb1f4f8a1431ae01f in term 0.
I20260808 03:57:23.821180 31766 leader_election.cc:302] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae10b034db5b43fc805fb1285d5e991c, c3f7d66592114addb1f4f8a1431ae01f; no voters:
I20260808 03:57:23.821635 32428 raft_consensus.cc:2781] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:57:23.821760 32428 raft_consensus.cc:482] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:57:23.821874 32428 raft_consensus.cc:3037] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:23.824371 32428 raft_consensus.cc:504] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.824981 32428 leader_election.cc:288] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [CANDIDATE]: Term 1 election: Requested vote from peers ae10b034db5b43fc805fb1285d5e991c (127.28.251.194:37739), 5bc8581e2a3e4e769d66bb45750f0b48 (127.28.251.195:43467)
I20260808 03:57:23.825582 32314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b22fd43994b441aa183f2531f118ea7" candidate_uuid: "c3f7d66592114addb1f4f8a1431ae01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc8581e2a3e4e769d66bb45750f0b48"
I20260808 03:57:23.825616 32093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b22fd43994b441aa183f2531f118ea7" candidate_uuid: "c3f7d66592114addb1f4f8a1431ae01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae10b034db5b43fc805fb1285d5e991c"
I20260808 03:57:23.825795 32314 raft_consensus.cc:3037] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:23.825834 32093 raft_consensus.cc:3037] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:23.827962 32314 raft_consensus.cc:2454] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f7d66592114addb1f4f8a1431ae01f in term 1.
I20260808 03:57:23.828411 31766 leader_election.cc:302] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bc8581e2a3e4e769d66bb45750f0b48, c3f7d66592114addb1f4f8a1431ae01f; no voters:
I20260808 03:57:23.828716 32428 raft_consensus.cc:2781] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:23.829087 32093 raft_consensus.cc:2454] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3f7d66592114addb1f4f8a1431ae01f in term 1.
I20260808 03:57:23.829219 32428 raft_consensus.cc:686] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 1 LEADER]: Becoming Leader. State: Replica: c3f7d66592114addb1f4f8a1431ae01f, State: Running, Role: LEADER
I20260808 03:57:23.829679 32428 consensus_queue.cc:237] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.833461 31618 catalog_manager.cc:5495] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f reported cstate change: term changed from 0 to 1, leader changed from <none> to c3f7d66592114addb1f4f8a1431ae01f (127.28.251.193). New cstate: current_term: 1 leader_uuid: "c3f7d66592114addb1f4f8a1431ae01f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } health_report { overall_health: UNKNOWN } } }
Aug 08 03:57:23 dist-test-slave-pww0 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161441, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:23.883092 32490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.865959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:42009 (local address 127.0.0.1:41614)
0808 03:57:23.867117 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:23.867148 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:23.867329 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:23.867955 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:23.867960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:23.868874 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:23.869169 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.869184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.870336 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.870342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.871923 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:23.871940 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.872725 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.872732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:23.872910 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.873676 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:23.873699 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:23.876594 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:23.879791 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:23.879807 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:23.879816 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:23.880519 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:23.880962 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:23.880967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:23.880971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:23.881210 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:23.881858 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:23.881867 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:23.882091 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 03:57:23.882308 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 03:57:23.882648 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"thread_start_us":818,"threads_started":1}
I20260808 03:57:23.883126 32492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:23.866058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.28.251.254:42009)
0808 03:57:23.867505 (+ 1447us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:23.867511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:23.867533 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:57:23.867579 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:23.867582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:23.867651 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:23.867751 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:23.869370 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.870161 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.872092 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:23.872527 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:23.872623 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:23.876844 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:23.876849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:23.876852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:23.876887 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:23.879607 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:23.880658 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:23.880662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:23.880665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:23.880830 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:23.881402 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:23.881405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:23.881407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:23.881700 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:23.881854 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:23.882425 (+ 571us) server_negotiation.cc:299] Negotiation successful
0808 03:57:23.882777 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":1283,"thread_start_us":1124,"threads_started":1}
I20260808 03:57:23.889330 31618 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:41614:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 03:57:23.890725 31618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:57:23.898737 31845 tablet_service.cc:1467] Processing CreateTablet for tablet 7e50f6394e774c18af464a49e87b5106 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03d57f562cc8480796c5141c6d4eae4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 03:57:23.899370 31845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e50f6394e774c18af464a49e87b5106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:23.899314 32277 tablet_service.cc:1467] Processing CreateTablet for tablet 7e50f6394e774c18af464a49e87b5106 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03d57f562cc8480796c5141c6d4eae4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 03:57:23.899353 32062 tablet_service.cc:1467] Processing CreateTablet for tablet 7e50f6394e774c18af464a49e87b5106 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03d57f562cc8480796c5141c6d4eae4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 03:57:23.899935 32277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e50f6394e774c18af464a49e87b5106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:23.899996 32062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e50f6394e774c18af464a49e87b5106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:23.907888 32421 tablet_bootstrap.cc:492] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c: Bootstrap starting.
I20260808 03:57:23.907871 32419 tablet_bootstrap.cc:492] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: Bootstrap starting.
I20260808 03:57:23.908337 32418 tablet_bootstrap.cc:492] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f: Bootstrap starting.
I20260808 03:57:23.909760 32419 tablet_bootstrap.cc:654] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:23.909778 32421 tablet_bootstrap.cc:654] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:23.910077 32418 tablet_bootstrap.cc:654] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:23.919253 32418 tablet_bootstrap.cc:492] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f: No bootstrap required, opened a new log
I20260808 03:57:23.919502 32418 ts_tablet_manager.cc:1397] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260808 03:57:23.919538 32419 tablet_bootstrap.cc:492] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: No bootstrap required, opened a new log
I20260808 03:57:23.919725 32419 ts_tablet_manager.cc:1397] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20260808 03:57:23.920432 32419 raft_consensus.cc:348] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.920394 32418 raft_consensus.cc:348] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.920645 32419 raft_consensus.cc:374] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:23.920722 32419 raft_consensus.cc:729] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc8581e2a3e4e769d66bb45750f0b48, State: Initialized, Role: FOLLOWER
I20260808 03:57:23.920650 32418 raft_consensus.cc:374] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:23.920809 32418 raft_consensus.cc:729] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f7d66592114addb1f4f8a1431ae01f, State: Initialized, Role: FOLLOWER
I20260808 03:57:23.920967 32419 consensus_queue.cc:260] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.921116 32418 consensus_queue.cc:260] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.921577 32428 tablet_replica.cc:439] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } } }
I20260808 03:57:23.921569 32427 tablet_replica.cc:439] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } } }
I20260808 03:57:23.921862 32428 tablet_replica.cc:442] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:23.921913 32427 tablet_replica.cc:442] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:23.922377 32419 ts_tablet_manager.cc:1428] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 03:57:23.924494 32421 tablet_bootstrap.cc:492] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c: No bootstrap required, opened a new log
I20260808 03:57:23.924757 32421 ts_tablet_manager.cc:1397] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.001s
I20260808 03:57:23.925580 32418 ts_tablet_manager.cc:1428] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260808 03:57:23.925585 32421 raft_consensus.cc:348] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.925850 32421 raft_consensus.cc:374] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:23.925940 32421 raft_consensus.cc:729] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae10b034db5b43fc805fb1285d5e991c, State: Initialized, Role: FOLLOWER
I20260808 03:57:23.926215 32421 consensus_queue.cc:260] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:23.926716 32459 tablet_replica.cc:439] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } } }
I20260808 03:57:23.927035 32459 tablet_replica.cc:442] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:23.928349 32421 ts_tablet_manager.cc:1428] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260808 03:57:24.089922 32427 raft_consensus.cc:482] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:24.090130 32427 raft_consensus.cc:504] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:24.091867 32427 leader_election.cc:288] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3f7d66592114addb1f4f8a1431ae01f (127.28.251.193:33963), ae10b034db5b43fc805fb1285d5e991c (127.28.251.194:37739)
I20260808 03:57:24.099123 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.092480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:59427 (local address 127.28.251.193:33963)
0808 03:57:24.093414 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:24.093420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:24.093442 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:57:24.093522 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:24.093527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:24.093605 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:24.093716 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:24.094240 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.095174 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.097178 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.098045 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.098233 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.098324 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:24.098647 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 03:57:24.098802 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":617,"threads_started":1}
I20260808 03:57:24.099231 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.092102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33963 (local address 127.28.251.195:59427)
0808 03:57:24.092751 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:24.092773 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:24.092904 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:24.093772 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:24.093777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:24.093814 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:24.094075 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:24.094084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.095327 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.095332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:24.096997 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:24.097012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.098234 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.098240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:24.098462 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.098509 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 03:57:24.098611 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 03:57:24.098766 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":2,"spinlock_wait_cycles":29568,"thread_start_us":423,"threads_started":1}
I20260808 03:57:24.099758 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.093241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:58675 (local address 127.28.251.194:37739)
0808 03:57:24.093455 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:24.093462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:24.093812 (+ 350us) server_negotiation.cc:407] Connection header received
0808 03:57:24.094097 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:24.094102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:24.094165 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:24.094248 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:24.094777 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.095518 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.097360 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.098213 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.098386 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.098462 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:24.098940 (+ 478us) server_negotiation.cc:299] Negotiation successful
0808 03:57:24.099092 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 03:57:24.100075 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e50f6394e774c18af464a49e87b5106" candidate_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f7d66592114addb1f4f8a1431ae01f" is_pre_election: true
I20260808 03:57:24.100423 31877 raft_consensus.cc:2454] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bc8581e2a3e4e769d66bb45750f0b48 in term 0.
I20260808 03:57:24.101104 32504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.093037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:37739 (local address 127.28.251.195:58675)
0808 03:57:24.093705 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:24.093723 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:24.093937 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:24.094286 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:24.094290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:24.094316 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:24.094601 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:24.094610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.095654 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.095657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:24.097193 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:24.097207 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.098456 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.098462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:24.098731 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.098774 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 03:57:24.100646 (+ 1872us) client_negotiation.cc:240] Negotiation successful
0808 03:57:24.100799 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":48,"thread_start_us":352,"threads_started":1}
I20260808 03:57:24.100987 32203 leader_election.cc:302] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bc8581e2a3e4e769d66bb45750f0b48, c3f7d66592114addb1f4f8a1431ae01f; no voters:
I20260808 03:57:24.101476 32427 raft_consensus.cc:2781] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:57:24.101596 32427 raft_consensus.cc:482] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:57:24.101711 32427 raft_consensus.cc:3037] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:24.101938 32093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e50f6394e774c18af464a49e87b5106" candidate_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae10b034db5b43fc805fb1285d5e991c" is_pre_election: true
I20260808 03:57:24.102190 32093 raft_consensus.cc:2454] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bc8581e2a3e4e769d66bb45750f0b48 in term 0.
I20260808 03:57:24.104614 32427 raft_consensus.cc:504] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:24.105360 32427 leader_election.cc:288] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [CANDIDATE]: Term 1 election: Requested vote from peers c3f7d66592114addb1f4f8a1431ae01f (127.28.251.193:33963), ae10b034db5b43fc805fb1285d5e991c (127.28.251.194:37739)
I20260808 03:57:24.105966 31877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e50f6394e774c18af464a49e87b5106" candidate_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3f7d66592114addb1f4f8a1431ae01f"
I20260808 03:57:24.105980 32093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e50f6394e774c18af464a49e87b5106" candidate_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae10b034db5b43fc805fb1285d5e991c"
I20260808 03:57:24.106186 31877 raft_consensus.cc:3037] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:24.106209 32093 raft_consensus.cc:3037] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:24.108456 31877 raft_consensus.cc:2454] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bc8581e2a3e4e769d66bb45750f0b48 in term 1.
I20260808 03:57:24.108466 32093 raft_consensus.cc:2454] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bc8581e2a3e4e769d66bb45750f0b48 in term 1.
I20260808 03:57:24.109066 32203 leader_election.cc:302] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bc8581e2a3e4e769d66bb45750f0b48, ae10b034db5b43fc805fb1285d5e991c; no voters:
I20260808 03:57:24.109458 32427 raft_consensus.cc:2781] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:24.109905 32427 raft_consensus.cc:686] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 1 LEADER]: Becoming Leader. State: Replica: 5bc8581e2a3e4e769d66bb45750f0b48, State: Running, Role: LEADER
I20260808 03:57:24.110391 32427 consensus_queue.cc:237] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } }
I20260808 03:57:24.112993 32512 tablet_replica.cc:439] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: TxnStatusTablet state changed. Reason: New leader 5bc8581e2a3e4e769d66bb45750f0b48. Latest consensus state: current_term: 1 leader_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } } }
I20260808 03:57:24.113389 32512 tablet_replica.cc:442] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: This TxnStatusTablet replica's current role is: LEADER
I20260808 03:57:24.114272 32515 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 03:57:24.114617 32515 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 03:57:24.114691 31616 catalog_manager.cc:5495] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bc8581e2a3e4e769d66bb45750f0b48 (127.28.251.195). New cstate: current_term: 1 leader_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } health_report { overall_health: HEALTHY } } }
I20260808 03:57:24.173924 32490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.165747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:43467 (local address 127.0.0.1:55880)
0808 03:57:24.166075 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:24.166100 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:24.166278 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:24.167119 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:24.167125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:24.167140 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 03:57:24.167355 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:24.167365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.168957 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.168963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:24.170633 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:24.170653 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.171862 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.171867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:24.172056 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.172131 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 03:57:24.173389 (+ 1258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 03:57:24.173419 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 03:57:24.173500 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 03:57:24.173651 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260808 03:57:24.174010 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.165752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.28.251.195:43467)
0808 03:57:24.166612 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:24.166618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:24.166642 (+ 24us) server_negotiation.cc:407] Connection header received
0808 03:57:24.166746 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:24.166751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:24.166912 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:24.167075 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:24.167551 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.168780 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.170843 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.171667 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.171819 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.172379 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:24.173239 (+ 860us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:24.173409 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:24.173570 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 03:57:24.173674 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710}
I20260808 03:57:24.179594 31877 raft_consensus.cc:1261] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f [term 1 FOLLOWER]: Refusing update from remote peer 5bc8581e2a3e4e769d66bb45750f0b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:24.179823 32093 raft_consensus.cc:1261] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c [term 1 FOLLOWER]: Refusing update from remote peer 5bc8581e2a3e4e769d66bb45750f0b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:24.180477 32427 consensus_queue.cc:1035] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:24.180861 32512 consensus_queue.cc:1035] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:24.184386 32428 tablet_replica.cc:439] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f: TxnStatusTablet state changed. Reason: New leader 5bc8581e2a3e4e769d66bb45750f0b48. Latest consensus state: current_term: 1 leader_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } } }
I20260808 03:57:24.184741 32428 tablet_replica.cc:442] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:24.188643 32519 mvcc.cc:205] Tried to move back new op lower bound from 7316117275351244800 to 7316117275080798208. Current Snapshot: MvccSnapshot[applied={T|T < 7316117275351244800}]
I20260808 03:57:24.189527 32512 tablet_replica.cc:439] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } } }
I20260808 03:57:24.189854 32512 tablet_replica.cc:442] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: This TxnStatusTablet replica's current role is: LEADER
I20260808 03:57:24.198832 32428 tablet_replica.cc:439] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } } }
I20260808 03:57:24.199174 32428 tablet_replica.cc:442] T 7e50f6394e774c18af464a49e87b5106 P c3f7d66592114addb1f4f8a1431ae01f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:24.204988 32459 tablet_replica.cc:439] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c: TxnStatusTablet state changed. Reason: New leader 5bc8581e2a3e4e769d66bb45750f0b48. Latest consensus state: current_term: 1 leader_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } } }
I20260808 03:57:24.205304 32459 tablet_replica.cc:442] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:24.208125 32427 tablet_replica.cc:439] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } } }
I20260808 03:57:24.208469 32427 tablet_replica.cc:442] T 7e50f6394e774c18af464a49e87b5106 P 5bc8581e2a3e4e769d66bb45750f0b48: This TxnStatusTablet replica's current role is: LEADER
I20260808 03:57:24.210502 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.201223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.28.251.193:33963)
0808 03:57:24.201420 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:24.201427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:24.202089 (+ 662us) server_negotiation.cc:407] Connection header received
0808 03:57:24.202329 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:24.202335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:24.202472 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:24.202665 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:24.203251 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.204335 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.206688 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.207392 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.207619 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.208345 (+ 726us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:24.209243 (+ 898us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:24.209396 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:24.210054 (+ 658us) server_negotiation.cc:299] Negotiation successful
0808 03:57:24.210154 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 03:57:24.210475 32520 mvcc.cc:205] Tried to move back new op lower bound from 7316117275351244800 to 7316117275080798208. Current Snapshot: MvccSnapshot[applied={T|T < 7316117275351244800}]
I20260808 03:57:24.213816 32459 tablet_replica.cc:439] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3f7d66592114addb1f4f8a1431ae01f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33963 } } peers { permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 } } peers { permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 } } }
I20260808 03:57:24.214109 32459 tablet_replica.cc:442] T 7e50f6394e774c18af464a49e87b5106 P ae10b034db5b43fc805fb1285d5e991c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:24.226543 32428 consensus_queue.cc:1035] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae10b034db5b43fc805fb1285d5e991c" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:24.230611 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.222874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:35573 (local address 127.28.251.195:43467)
0808 03:57:24.223061 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:24.223066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:24.223715 (+ 649us) server_negotiation.cc:407] Connection header received
0808 03:57:24.223933 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:24.223939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:24.224021 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:24.224102 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:24.224724 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.225684 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.228054 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.228833 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.228943 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.229706 (+ 763us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:24.229859 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:24.229975 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:24.230138 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 03:57:24.230237 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 03:57:24.230720 32536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.222923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:43467 (local address 127.28.251.193:35573)
0808 03:57:24.223603 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:24.223621 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:24.223788 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:24.224140 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:24.224144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:24.224156 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 03:57:24.224546 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:24.224560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.225900 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.225906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:24.227859 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:24.227878 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.229157 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.229165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:24.229372 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.229456 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 03:57:24.229998 (+ 542us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 03:57:24.230010 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 03:57:24.230095 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 03:57:24.230275 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":326,"threads_started":1}
I20260808 03:57:24.234437 32428 consensus_queue.cc:1035] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bc8581e2a3e4e769d66bb45750f0b48" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:24.280767 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.273202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:50309 (local address 127.28.251.193:33963)
0808 03:57:24.273409 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:24.273415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:24.274351 (+ 936us) server_negotiation.cc:407] Connection header received
0808 03:57:24.274619 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:24.274626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:24.274731 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:24.274826 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:24.275611 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.276404 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.278718 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.279383 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.279539 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.280076 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:24.280201 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:24.280273 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:24.280425 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 03:57:24.280495 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 03:57:24.281032 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.273244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33963 (local address 127.28.251.195:50309)
0808 03:57:24.274244 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:24.274269 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:24.274472 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:24.274999 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:24.275005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:24.275018 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 03:57:24.275398 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:24.275410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.276550 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.276556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:24.278499 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:24.278518 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.279605 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.279612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:24.279812 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.279895 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 03:57:24.280307 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 03:57:24.280321 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 03:57:24.280404 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 03:57:24.280563 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":575,"threads_started":1}
I20260808 03:57:24.324360 31616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:24.324661 31616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:24.328474 31616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5ad389c0e0247df9a19715375ba140d: Sending DeleteTablet for 3 replicas of tablet 7b22fd43994b441aa183f2531f118ea7
I20260808 03:57:24.329576 32062 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b22fd43994b441aa183f2531f118ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:24 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:55426
I20260808 03:57:24.329895 31845 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b22fd43994b441aa183f2531f118ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:24 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:51860
I20260808 03:57:24.329912 32277 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b22fd43994b441aa183f2531f118ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:24 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:55866
I20260808 03:57:24.340005 32548 tablet_replica.cc:330] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48: stopping tablet replica
I20260808 03:57:24.340248 32548 raft_consensus.cc:2229] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:24.340536 32548 raft_consensus.cc:2258] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:24.344427 32548 ts_tablet_manager.cc:1905] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:24.353416 32548 ts_tablet_manager.cc:1918] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:57:24.353683 32548 log.cc:1196] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/7b22fd43994b441aa183f2531f118ea7
I20260808 03:57:24.354293 32548 ts_tablet_manager.cc:1939] T 7b22fd43994b441aa183f2531f118ea7 P 5bc8581e2a3e4e769d66bb45750f0b48: Deleting consensus metadata
I20260808 03:57:24.355321 32551 tablet_replica.cc:330] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f: stopping tablet replica
I20260808 03:57:24.355580 32551 raft_consensus.cc:2229] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:57:24.355715 31584 catalog_manager.cc:4841] TS 5bc8581e2a3e4e769d66bb45750f0b48 (127.28.251.195:43467): tablet 7b22fd43994b441aa183f2531f118ea7 (table test-table [id=5aa666bea02d41be9c3af5edb4499fb0]) successfully deleted
I20260808 03:57:24.356238 32551 raft_consensus.cc:2258] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:24.357581 32547 tablet_replica.cc:330] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c: stopping tablet replica
I20260808 03:57:24.357885 32547 raft_consensus.cc:2229] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:24.358067 32551 ts_tablet_manager.cc:1905] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:24.358268 32547 raft_consensus.cc:2258] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:24.366225 32551 ts_tablet_manager.cc:1918] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:57:24.366410 32551 log.cc:1196] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/7b22fd43994b441aa183f2531f118ea7
I20260808 03:57:24.366967 32551 ts_tablet_manager.cc:1939] T 7b22fd43994b441aa183f2531f118ea7 P c3f7d66592114addb1f4f8a1431ae01f: Deleting consensus metadata
I20260808 03:57:24.368397 31584 catalog_manager.cc:4841] TS c3f7d66592114addb1f4f8a1431ae01f (127.28.251.193:33963): tablet 7b22fd43994b441aa183f2531f118ea7 (table test-table [id=5aa666bea02d41be9c3af5edb4499fb0]) successfully deleted
Aug 08 03:57:24 dist-test-slave-pww0 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:57:24.370285 32547 ts_tablet_manager.cc:1905] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:24.380388 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.353237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.28.251.193:33963)
0808 03:57:24.353468 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:24.353476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:24.356955 (+ 3479us) server_negotiation.cc:407] Connection header received
0808 03:57:24.357179 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:24.357185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:24.357289 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:24.357395 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:24.360930 (+ 3535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.362119 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.364183 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.364961 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.365066 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.371389 (+ 6323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:24.371418 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:24.371441 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:24.371558 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:24.375967 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:24.377091 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:24.377103 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:24.377106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:24.377291 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:24.377805 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:24.377811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:24.377814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:24.378430 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:24.378770 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:24.379425 (+ 655us) server_negotiation.cc:299] Negotiation successful
0808 03:57:24.380011 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260808 03:57:24.382042 31949 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51868
I20260808 03:57:24.388460 32547 ts_tablet_manager.cc:1918] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:57:24.388635 32547 log.cc:1196] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/7b22fd43994b441aa183f2531f118ea7
I20260808 03:57:24.389114 32547 ts_tablet_manager.cc:1939] T 7b22fd43994b441aa183f2531f118ea7 P ae10b034db5b43fc805fb1285d5e991c: Deleting consensus metadata
I20260808 03:57:24.390548 31584 catalog_manager.cc:4841] TS ae10b034db5b43fc805fb1285d5e991c (127.28.251.194:37739): tablet 7b22fd43994b441aa183f2531f118ea7 (table test-table [id=5aa666bea02d41be9c3af5edb4499fb0]) successfully deleted
I20260808 03:57:24.428660 32560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.392199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.28.251.254:42009)
0808 03:57:24.411620 (+ 19421us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:24.411627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:24.411653 (+ 26us) server_negotiation.cc:407] Connection header received
0808 03:57:24.411722 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:24.411727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:24.411815 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:24.411953 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:24.414075 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.415506 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.417511 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.418173 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.418316 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.421500 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:24.421507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:24.421513 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:24.421560 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:24.425483 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:24.426382 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:24.426388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:24.426391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:24.426556 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:24.427001 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:24.427005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:24.427009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:24.427391 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:24.427697 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:24.428031 (+ 334us) server_negotiation.cc:299] Negotiation successful
0808 03:57:24.428302 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19221,"thread_start_us":20782,"threads_started":1}
W20260808 03:57:24.429347 31616 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41620
I20260808 03:57:24.432284 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31736
I20260808 03:57:24.453133 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.433575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.28.251.193:33963)
0808 03:57:24.433782 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:24.433789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:24.435967 (+ 2178us) server_negotiation.cc:407] Connection header received
0808 03:57:24.436124 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:24.436129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:24.436216 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:24.436321 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:24.437891 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.439160 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.440986 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.441710 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.441857 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.444955 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:24.444962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:24.444968 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:24.445013 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:24.449197 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:24.450174 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:24.450181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:24.450185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:24.450383 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:24.450892 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:24.450899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:24.450902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:24.451318 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:24.451694 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:24.452199 (+ 505us) server_negotiation.cc:299] Negotiation successful
0808 03:57:24.452497 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 03:57:24.454165 31949 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51876
W20260808 03:57:24.454924 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31736: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 03:57:24.455132 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 31736
I20260808 03:57:24.486581 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31962
Aug 08 03:57:24 dist-test-slave-pww0 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 03:57:24.503149 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.487433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.28.251.194:37739)
0808 03:57:24.487645 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:24.487651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:24.487743 (+ 92us) server_negotiation.cc:407] Connection header received
0808 03:57:24.487970 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:24.487974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:24.488041 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:24.488126 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:24.489357 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.490069 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.491378 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.491931 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.492055 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.496661 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:24.496684 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:24.496701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:24.496750 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:24.499910 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:24.500907 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:24.500916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:24.500918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:24.501074 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:24.501525 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:24.501528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:24.501530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:24.501966 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:24.502229 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:24.502636 (+ 407us) server_negotiation.cc:299] Negotiation successful
0808 03:57:24.502837 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260808 03:57:24.503857 32168 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55450
W20260808 03:57:24.504541 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31962: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 03:57:24.504714 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 31962
I20260808 03:57:24.543051 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 32181
Aug 08 03:57:24 dist-test-slave-pww0 krb5kdc[31545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 03:57:24.559643 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:24.543862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.28.251.195:43467)
0808 03:57:24.544065 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:24.544071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:24.544132 (+ 61us) server_negotiation.cc:407] Connection header received
0808 03:57:24.544357 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:24.544362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:24.544441 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:24.544540 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:24.546051 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.547009 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.548410 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:24.548840 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:24.548955 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:24.553332 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:24.553354 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:24.553374 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:24.553418 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:24.556539 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:24.557308 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:24.557316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:24.557318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:24.557486 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:24.557920 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:24.557925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:24.557929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:24.558370 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:24.558733 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:24.559007 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 03:57:24.559238 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260808 03:57:24.560369 32383 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55890
W20260808 03:57:24.561079 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 32181: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 03:57:24.561223 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 32181
I20260808 03:57:24.596117 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31561
I20260808 03:57:24.597317 31681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:24.717268 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 31561
2026-08-08T03:57:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5338 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:57:24 dist-test-slave-pww0 krb5kdc[32574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:57:24 dist-test-slave-pww0 krb5kdc[32574](info): set up 2 sockets
Aug 08 03:57:24 dist-test-slave-pww0 krb5kdc[32574](info): commencing operation
krb5kdc: starting...
W20260808 03:57:25.974267 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:57:26 dist-test-slave-pww0 krb5kdc[32574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:57:26.221150 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43961
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:43961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:26.567093 32599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:26.567426 32599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:26.567600 32599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:26.575193 32599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:26.575336 32599 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 03:57:26.575412 32599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:26.575531 32599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:26.575600 32599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:26.575700 32599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:57:26.586225 32599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:43961
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:26.587689 32599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:26.589107 32599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:26.601707 32616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:26.601776 32608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:26.601756 32611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:26.602983 32599 server_base.cc:1034] running on GCE node
I20260808 03:57:26.603888 32599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:26.605360 32599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:26.606606 32599 hybrid_clock.cc:648] HybridClock initialized: now 1786161446606528 us; error 42 us; skew 500 ppm
I20260808 03:57:26.606978 32599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:26 dist-test-slave-pww0 krb5kdc[32574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:26.610702 32599 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:57:26.614271 32599 webserver.cc:460] Webserver started at http://127.28.251.254:45791/ using document root <none> and password file <none>
I20260808 03:57:26.615614 32599 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:26.615774 32599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:26.616380 32599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:26.621567 32599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "865f5d73f42944589e531a509d46d3fc"
format_stamp: "Formatted at 2026-08-08 03:57:26 on dist-test-slave-pww0"
server_key: "e17c94318ff62e6dc99fd79b104ce2dd"
server_key_iv: "854a5720930ea3f2ac3a47404d3f4791"
server_key_version: "encryptionkey@0"
I20260808 03:57:26.622874 32599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "865f5d73f42944589e531a509d46d3fc"
format_stamp: "Formatted at 2026-08-08 03:57:26 on dist-test-slave-pww0"
server_key: "e17c94318ff62e6dc99fd79b104ce2dd"
server_key_iv: "854a5720930ea3f2ac3a47404d3f4791"
server_key_version: "encryptionkey@0"
I20260808 03:57:26.630591 32599 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260808 03:57:26.635797 32629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:26.637712 32599 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 03:57:26.637980 32599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "865f5d73f42944589e531a509d46d3fc"
format_stamp: "Formatted at 2026-08-08 03:57:26 on dist-test-slave-pww0"
server_key: "e17c94318ff62e6dc99fd79b104ce2dd"
server_key_iv: "854a5720930ea3f2ac3a47404d3f4791"
server_key_version: "encryptionkey@0"
I20260808 03:57:26.638240 32599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:26.682417 32599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:26.689327 32599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:26.689675 32599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:26.710651 32599 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:43961
I20260808 03:57:26.710680 32746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:43961 every 8 connection(s)
I20260808 03:57:26.712159 32599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:26.715921 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 32599
I20260808 03:57:26.716112 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:26.716472 32747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:26.716799 29679 external_mini_cluster.cc:1428] Setting key cb56be1ba5dc0447e3b5fdb13a66c8f7
I20260808 03:57:26.731783 32747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc: Bootstrap starting.
Aug 08 03:57:26 dist-test-slave-pww0 krb5kdc[32574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:26.735842 32747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:26.737370 32747 log.cc:824] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:26.741719 32747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc: No bootstrap required, opened a new log
I20260808 03:57:26.742668 32750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:26.718957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.28.251.254:43961)
0808 03:57:26.719923 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:26.719936 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:26.719989 (+ 53us) server_negotiation.cc:407] Connection header received
0808 03:57:26.721589 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:26.721617 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:26.722043 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:26.722425 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:26.724413 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:26.725766 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:26.727649 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:26.728440 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:26.728594 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:26.733713 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:26.733726 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:26.733747 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:26.733807 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:26.737760 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:26.738781 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:26.738795 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:26.738804 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:26.739089 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:26.739605 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:26.739611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:26.739614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:26.740211 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:26.740461 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:26.741062 (+ 601us) server_negotiation.cc:299] Negotiation successful
0808 03:57:26.741502 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":447,"threads_started":1}
I20260808 03:57:26.750219 32747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865f5d73f42944589e531a509d46d3fc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43961 } }
I20260808 03:57:26.750674 32747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:26.750814 32747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865f5d73f42944589e531a509d46d3fc, State: Initialized, Role: FOLLOWER
I20260808 03:57:26.751561 32747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865f5d73f42944589e531a509d46d3fc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43961 } }
I20260808 03:57:26.751905 32747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:26.752074 32747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:26.752249 32747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:26.755236 32747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865f5d73f42944589e531a509d46d3fc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43961 } }
I20260808 03:57:26.755921 32747 leader_election.cc:302] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 865f5d73f42944589e531a509d46d3fc; no voters:
I20260808 03:57:26.756568 32747 leader_election.cc:288] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:26.756722 32754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:26.757480 32754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [term 1 LEADER]: Becoming Leader. State: Replica: 865f5d73f42944589e531a509d46d3fc, State: Running, Role: LEADER
I20260808 03:57:26.758036 32754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865f5d73f42944589e531a509d46d3fc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43961 } }
I20260808 03:57:26.758757 32747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:26.761059 32756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 865f5d73f42944589e531a509d46d3fc. Latest consensus state: current_term: 1 leader_uuid: "865f5d73f42944589e531a509d46d3fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865f5d73f42944589e531a509d46d3fc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43961 } } }
I20260808 03:57:26.761165 32755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "865f5d73f42944589e531a509d46d3fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865f5d73f42944589e531a509d46d3fc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43961 } } }
I20260808 03:57:26.761561 32756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:26.761559 32755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:26.762564 32767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:26.768561 32767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:26.778800 32767 catalog_manager.cc:1347] Generated new cluster ID: 2939cfefa4d742f2a29a63d0251180cf
I20260808 03:57:26.778935 32767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:26.790261 32767 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:26.791893 32767 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:26.802018 32767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc: Generated new TSK 0
I20260808 03:57:26.802948 32767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:57:26.863952 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43961
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:27.220095 325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:27.220415 325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:27.220556 325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:27.228204 325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:27.228335 325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:27.228454 325 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 03:57:27.228521 325 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 03:57:27.228611 325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:57:27.238920 325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.251.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:27.240453 325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:27.242010 325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:27.255875 332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:27.255945 337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:27.255924 333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:27.256598 325 server_base.cc:1034] running on GCE node
I20260808 03:57:27.257192 325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:27.258162 325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:27.259405 325 hybrid_clock.cc:648] HybridClock initialized: now 1786161447259331 us; error 48 us; skew 500 ppm
I20260808 03:57:27.259883 325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:27 dist-test-slave-pww0 krb5kdc[32574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161447, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:27.267521 325 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:57:27.269270 325 webserver.cc:460] Webserver started at http://127.28.251.193:34363/ using document root <none> and password file <none>
I20260808 03:57:27.270040 325 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:27.270167 325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:27.270526 325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:27.273747 325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "0a21c0cfa099427890706cc2babfbe5c"
format_stamp: "Formatted at 2026-08-08 03:57:27 on dist-test-slave-pww0"
server_key: "fc5bef5c764117f56d4fec9837daa5d6"
server_key_iv: "6912aeac1e6961db6074a72caab26c7e"
server_key_version: "encryptionkey@0"
I20260808 03:57:27.274555 325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "0a21c0cfa099427890706cc2babfbe5c"
format_stamp: "Formatted at 2026-08-08 03:57:27 on dist-test-slave-pww0"
server_key: "fc5bef5c764117f56d4fec9837daa5d6"
server_key_iv: "6912aeac1e6961db6074a72caab26c7e"
server_key_version: "encryptionkey@0"
I20260808 03:57:27.280124 325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 03:57:27.283897 346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:27.285379 325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:57:27.285555 325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "0a21c0cfa099427890706cc2babfbe5c"
format_stamp: "Formatted at 2026-08-08 03:57:27 on dist-test-slave-pww0"
server_key: "fc5bef5c764117f56d4fec9837daa5d6"
server_key_iv: "6912aeac1e6961db6074a72caab26c7e"
server_key_version: "encryptionkey@0"
I20260808 03:57:27.285768 325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:27.306229 325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:27.311153 325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:27.311517 325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:27.314745 325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:27.314900 325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:27.315096 325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:27.315204 325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 03:57:27 dist-test-slave-pww0 krb5kdc[32574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161447, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:27.341130 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:27.315275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:47007 (local address 127.28.251.254:43961)
0808 03:57:27.316231 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:27.316236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:27.317284 (+ 1048us) server_negotiation.cc:407] Connection header received
0808 03:57:27.319740 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:27.319746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:27.319855 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:27.319961 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:27.322784 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.323851 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.326483 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.327059 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.327210 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:27.332823 (+ 5613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:27.332832 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:27.332837 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:27.332888 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:27.336518 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:27.337883 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:27.337890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:27.337894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:27.338075 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:27.338771 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:27.338777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:27.338779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:27.339066 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:27.339216 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:27.340361 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0808 03:57:27.340663 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":720,"threads_started":1}
I20260808 03:57:27.342000 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:27.316140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43961 (local address 127.28.251.193:47007)
0808 03:57:27.317093 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:27.317137 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:27.319376 (+ 2239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:27.320320 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:27.320333 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:27.321227 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:27.322384 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:27.322453 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.324050 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.324057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:27.326293 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:27.326310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.327357 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.327367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:27.327733 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:27.328644 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:27.328687 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:27.332456 (+ 3769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:27.336780 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:27.336794 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:27.336822 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:27.337697 (+ 875us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:27.338221 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:27.338230 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:27.338235 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:27.338628 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:27.339224 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:27.339238 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:27.339865 (+ 627us) client_negotiation.cc:769] Sending connection context
0808 03:57:27.340242 (+ 377us) client_negotiation.cc:240] Negotiation successful
0808 03:57:27.340743 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":391,"threads_started":1}
I20260808 03:57:27.364246 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:27.348974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:59249 (local address 127.28.251.254:43961)
0808 03:57:27.349229 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:27.349237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:27.349765 (+ 528us) server_negotiation.cc:407] Connection header received
0808 03:57:27.350076 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:27.350081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:27.350158 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:27.350255 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:27.351888 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.352868 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.355146 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.355732 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.355879 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:27.358144 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:27.358150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:27.358153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:27.358193 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:27.361202 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:27.361940 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:27.361945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:27.361948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:27.362090 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:27.362478 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:27.362482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:27.362484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:27.362782 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:27.362942 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:27.363526 (+ 584us) server_negotiation.cc:299] Negotiation successful
0808 03:57:27.363830 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 03:57:27.364470 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:27.348930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43961 (local address 127.28.251.193:59249)
0808 03:57:27.349618 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:27.349648 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:27.349838 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:27.350308 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:27.350314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:27.351022 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:27.351638 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:27.351659 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.353055 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.353063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:27.354898 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:27.354924 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.355982 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.355989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:27.356184 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:27.356972 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:27.356992 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:27.357894 (+ 902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:27.361364 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:27.361369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:27.361372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:27.361802 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:27.362220 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:27.362224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:27.362227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:27.362361 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:27.362918 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:27.362923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:27.363150 (+ 227us) client_negotiation.cc:769] Sending connection context
0808 03:57:27.363316 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 03:57:27.363786 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":331,"threads_started":1}
I20260808 03:57:27.370440 325 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:35981
I20260808 03:57:27.370496 554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:35981 every 8 connection(s)
I20260808 03:57:27.371811 325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:57:27.374295 555 heartbeater.cc:347] Connected to a master server at 127.28.251.254:43961
I20260808 03:57:27.374536 555 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:27.375283 555 heartbeater.cc:511] Master 127.28.251.254:43961 requested a full tablet report, sending...
I20260808 03:57:27.377692 32667 ts_manager.cc:194] Registered new tserver with Master: 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981)
I20260808 03:57:27.379240 32667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:47007
I20260808 03:57:27.379999 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 325
I20260808 03:57:27.380247 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:57:27.380970 29679 external_mini_cluster.cc:1428] Setting key d671c5765c6b3ddf4765c6b21df08ffc
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:57:27.433200 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43961
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:27.782944 565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:27.783378 565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:27.783562 565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:27.791429 565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:27.791612 565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:27.791749 565 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 03:57:27.791821 565 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 03:57:27.791927 565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:57:27.802502 565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.251.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:27.803992 565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:27.805456 565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:27.819130 571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:27.819219 576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:27.819160 570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:27.820150 565 server_base.cc:1034] running on GCE node
I20260808 03:57:27.820842 565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:27.821864 565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:27.823104 565 hybrid_clock.cc:648] HybridClock initialized: now 1786161447823050 us; error 40 us; skew 500 ppm
I20260808 03:57:27.823477 565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:27 dist-test-slave-pww0 krb5kdc[32574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161447, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:27.826937 565 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:57:27.828567 565 webserver.cc:460] Webserver started at http://127.28.251.194:33129/ using document root <none> and password file <none>
I20260808 03:57:27.829294 565 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:27.829418 565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:27.829770 565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:27.832984 565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "3ae8ed3767384f0eb0bd7f8d36890390"
format_stamp: "Formatted at 2026-08-08 03:57:27 on dist-test-slave-pww0"
server_key: "3faaae0e4779ad7b5f70d2bb25bd0373"
server_key_iv: "b55f80c50f8587df85e916df48f35b0f"
server_key_version: "encryptionkey@0"
I20260808 03:57:27.833787 565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "3ae8ed3767384f0eb0bd7f8d36890390"
format_stamp: "Formatted at 2026-08-08 03:57:27 on dist-test-slave-pww0"
server_key: "3faaae0e4779ad7b5f70d2bb25bd0373"
server_key_iv: "b55f80c50f8587df85e916df48f35b0f"
server_key_version: "encryptionkey@0"
I20260808 03:57:27.839586 565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 03:57:27.843528 589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:27.845247 565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:57:27.845422 565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "3ae8ed3767384f0eb0bd7f8d36890390"
format_stamp: "Formatted at 2026-08-08 03:57:27 on dist-test-slave-pww0"
server_key: "3faaae0e4779ad7b5f70d2bb25bd0373"
server_key_iv: "b55f80c50f8587df85e916df48f35b0f"
server_key_version: "encryptionkey@0"
I20260808 03:57:27.845688 565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:27.869419 565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:27.874081 565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:27.874409 565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:27.877452 565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:27.877585 565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:27.877707 565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:27.877794 565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 03:57:27 dist-test-slave-pww0 krb5kdc[32574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161447, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:27.904357 606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:27.878986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43961 (local address 127.28.251.194:49525)
0808 03:57:27.879923 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:27.879967 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:27.881893 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:27.882771 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:27.882788 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:27.883669 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:27.884645 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:27.884689 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.886072 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.886078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:27.888730 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:27.888754 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.890116 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.890127 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:27.890558 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:27.891418 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:27.891479 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:27.894732 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:27.898599 (+ 3867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:27.898612 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:27.898632 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:27.899388 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:27.899963 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:27.899972 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:27.899975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:27.900311 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:27.900990 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:27.900998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:27.901585 (+ 587us) client_negotiation.cc:769] Sending connection context
0808 03:57:27.901955 (+ 370us) client_negotiation.cc:240] Negotiation successful
0808 03:57:27.902612 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":402,"threads_started":1}
I20260808 03:57:27.905486 604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:27.878139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:49525 (local address 127.28.251.254:43961)
0808 03:57:27.878755 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:27.878760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:27.880097 (+ 1337us) server_negotiation.cc:407] Connection header received
0808 03:57:27.882191 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:27.882197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:27.882293 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:27.882392 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:27.884884 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.885838 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.888967 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.889620 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.889780 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:27.895024 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:27.895031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:27.895036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:27.895085 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:27.898433 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:27.899652 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:27.899659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:27.899663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:27.899811 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:27.900451 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:27.900457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:27.900460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:27.900849 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:27.901006 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:27.904992 (+ 3986us) server_negotiation.cc:299] Negotiation successful
0808 03:57:27.905212 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":340,"threads_started":1}
I20260808 03:57:27.925421 565 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:41777
I20260808 03:57:27.925444 805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:41777 every 8 connection(s)
I20260808 03:57:27.926239 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:27.911423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43961 (local address 127.28.251.194:59853)
0808 03:57:27.912387 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:27.912409 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:27.912590 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:27.912939 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:27.912944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:27.913443 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:27.913752 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:27.913763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.914806 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.914810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:27.916096 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:27.916112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.917146 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.917151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:27.917310 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:27.918002 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:27.918041 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:27.919139 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:27.923383 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:27.923388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:27.923391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:27.923853 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:27.924325 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:27.924331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:27.924334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:27.924524 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:27.925185 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:27.925190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:27.925415 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 03:57:27.925580 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 03:57:27.925891 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":600,"threads_started":1}
I20260808 03:57:27.926967 565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:57:27.926996 604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:27.911573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:59853 (local address 127.28.251.254:43961)
0808 03:57:27.911819 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:27.911827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:27.912502 (+ 675us) server_negotiation.cc:407] Connection header received
0808 03:57:27.912729 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:27.912734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:27.912803 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:27.912888 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:27.913913 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.914650 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.916295 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:27.916935 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:27.917088 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:27.919512 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:27.919520 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:27.919524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:27.919574 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:27.923213 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:27.923991 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:27.923998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:27.924001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:27.924178 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:27.924687 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:27.924691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:27.924693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:27.925046 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:27.925192 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:27.926570 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0808 03:57:27.926769 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 03:57:27.927927 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 565
I20260808 03:57:27.928120 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:57:27.928726 29679 external_mini_cluster.cc:1428] Setting key 158084246d538751755af8910f972959
I20260808 03:57:27.929430 810 heartbeater.cc:347] Connected to a master server at 127.28.251.254:43961
I20260808 03:57:27.929668 810 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:27.930440 810 heartbeater.cc:511] Master 127.28.251.254:43961 requested a full tablet report, sending...
I20260808 03:57:27.932451 32667 ts_manager.cc:194] Registered new tserver with Master: 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194:41777)
I20260808 03:57:27.933257 32667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:49525
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:57:27.982836 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43961
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:28.345428 817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:28.345777 817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:28.345942 817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:28.353940 817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:28.354091 817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:28.354235 817 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 03:57:28.354310 817 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 03:57:28.354431 817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:57:28.365269 817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.251.254:43961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:28.366742 817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:28.368232 817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 03:57:28.382773 555 heartbeater.cc:503] Master 127.28.251.254:43961 was elected leader, sending a full tablet report...
W20260808 03:57:28.383064 825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:28.382961 826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:28.383846 828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:28.383813 817 server_base.cc:1034] running on GCE node
I20260808 03:57:28.384747 817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:28.385857 817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:28.387095 817 hybrid_clock.cc:648] HybridClock initialized: now 1786161448387024 us; error 47 us; skew 500 ppm
I20260808 03:57:28.387481 817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:28 dist-test-slave-pww0 krb5kdc[32574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:28.390820 817 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:57:28.392514 817 webserver.cc:460] Webserver started at http://127.28.251.195:41339/ using document root <none> and password file <none>
I20260808 03:57:28.393260 817 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:28.393414 817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:28.393752 817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:28.397352 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "1f52ac8f849f4a07b6555ad0ba250230"
format_stamp: "Formatted at 2026-08-08 03:57:28 on dist-test-slave-pww0"
server_key: "4122b4c38e1f131c0d4e51f6e2d7a238"
server_key_iv: "667677cb1259121d8feb69b08839f093"
server_key_version: "encryptionkey@0"
I20260808 03:57:28.398185 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "1f52ac8f849f4a07b6555ad0ba250230"
format_stamp: "Formatted at 2026-08-08 03:57:28 on dist-test-slave-pww0"
server_key: "4122b4c38e1f131c0d4e51f6e2d7a238"
server_key_iv: "667677cb1259121d8feb69b08839f093"
server_key_version: "encryptionkey@0"
I20260808 03:57:28.404079 817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 03:57:28.408016 841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:28.409466 817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:57:28.409651 817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "1f52ac8f849f4a07b6555ad0ba250230"
format_stamp: "Formatted at 2026-08-08 03:57:28 on dist-test-slave-pww0"
server_key: "4122b4c38e1f131c0d4e51f6e2d7a238"
server_key_iv: "667677cb1259121d8feb69b08839f093"
server_key_version: "encryptionkey@0"
I20260808 03:57:28.409862 817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:28.425064 817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:28.429620 817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:28.429930 817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:28.432958 817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:28.433072 817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:28.433197 817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:28.433300 817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 03:57:28 dist-test-slave-pww0 krb5kdc[32574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:28.457620 858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.433592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:57739 (local address 127.28.251.254:43961)
0808 03:57:28.434221 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:28.434228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:28.435404 (+ 1176us) server_negotiation.cc:407] Connection header received
0808 03:57:28.437606 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:28.437612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:28.437716 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:28.437815 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:28.440056 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.441125 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.443735 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.444255 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.444393 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.449955 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:28.449961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:28.449964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:28.450010 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:28.453347 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:28.454582 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:28.454589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:28.454593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:28.454783 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:28.455561 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:28.455567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:28.455571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:28.455980 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:28.456127 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:28.456966 (+ 839us) server_negotiation.cc:299] Negotiation successful
0808 03:57:28.457215 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":364,"threads_started":1}
I20260808 03:57:28.458841 860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.434276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43961 (local address 127.28.251.195:57739)
0808 03:57:28.435178 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:28.435240 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:28.437297 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:28.438102 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:28.438111 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:28.438871 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:28.439833 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.439864 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.441298 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.441304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.443576 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.443592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.444568 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.444582 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.444973 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.445890 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:28.445926 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:28.449554 (+ 3628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:28.453534 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:28.453551 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:28.453574 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:28.454422 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:28.454943 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:28.454949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:28.454953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:28.455359 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:28.456138 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:28.456147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:28.456555 (+ 408us) client_negotiation.cc:769] Sending connection context
0808 03:57:28.456825 (+ 270us) client_negotiation.cc:240] Negotiation successful
0808 03:57:28.457437 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":358,"threads_started":1}
I20260808 03:57:28.478992 817 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:44511
I20260808 03:57:28.479031 1041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:44511 every 8 connection(s)
I20260808 03:57:28.480505 817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:57:28.482857 1042 heartbeater.cc:347] Connected to a master server at 127.28.251.254:43961
I20260808 03:57:28.483116 1042 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:28.483520 858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.466570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:44927 (local address 127.28.251.254:43961)
0808 03:57:28.466820 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:28.466828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:28.467446 (+ 618us) server_negotiation.cc:407] Connection header received
0808 03:57:28.467717 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:28.467722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:28.467818 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:28.467933 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:28.469742 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.470586 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.472920 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.473548 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.473711 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.476765 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:28.476772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:28.476775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:28.476816 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:28.480030 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:28.480934 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:28.480939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:28.480942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:28.481106 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:28.481656 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:28.481662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:28.481665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:28.482103 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:28.482266 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:28.482851 (+ 585us) server_negotiation.cc:299] Negotiation successful
0808 03:57:28.483087 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 03:57:28.483655 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.466619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43961 (local address 127.28.251.195:44927)
0808 03:57:28.467343 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:28.467360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:28.467551 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:28.468172 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:28.468178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:28.468978 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:28.469523 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.469542 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.470818 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.470825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.472643 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.472671 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.474033 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.474043 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.474269 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.475070 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:28.475115 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:28.476454 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:28.480212 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:28.480219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:28.480224 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:28.480783 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:28.481248 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:28.481254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:28.481259 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:28.481463 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:28.482357 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:28.482361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:28.482542 (+ 181us) client_negotiation.cc:769] Sending connection context
0808 03:57:28.482697 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 03:57:28.483111 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":380,"threads_started":1}
I20260808 03:57:28.484082 1042 heartbeater.cc:511] Master 127.28.251.254:43961 requested a full tablet report, sending...
I20260808 03:57:28.485610 32667 ts_manager.cc:194] Registered new tserver with Master: 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195:44511)
I20260808 03:57:28.486390 32667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:57739
I20260808 03:57:28.488545 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 817
I20260808 03:57:28.488727 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:57:28.489274 29679 external_mini_cluster.cc:1428] Setting key 6b089ee9a435393627647bdcc8fd8812
I20260808 03:57:28.492482 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 03:57:28 dist-test-slave-pww0 krb5kdc[32574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 03:57:28 dist-test-slave-pww0 krb5kdc[32574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:28.550966 858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.536753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.28.251.254:43961)
0808 03:57:28.536938 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:28.536943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:28.537202 (+ 259us) server_negotiation.cc:407] Connection header received
0808 03:57:28.537395 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:28.537398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:28.537466 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:28.537561 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:28.538853 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.539672 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.540912 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.541316 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.541416 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.545397 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:28.545402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:28.545406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:28.545445 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:28.548217 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:28.548991 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:28.548995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:28.548998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:28.549190 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:28.549596 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:28.549601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:28.549603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:28.549919 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:28.550143 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:28.550455 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 03:57:28.550677 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 03:57:28.555601 32667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:57:28.562780 32667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:57:28.601222 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.576463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.28.251.193:35981)
0808 03:57:28.591847 (+ 15384us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:28.591859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:28.591895 (+ 36us) server_negotiation.cc:407] Connection header received
0808 03:57:28.592011 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:28.592021 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:28.592331 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:28.592572 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:28.593361 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.595131 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.598114 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.599544 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.599739 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.599926 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:28.600665 (+ 739us) server_negotiation.cc:299] Negotiation successful
0808 03:57:28.600872 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15168,"thread_start_us":14986,"threads_started":1}
I20260808 03:57:28.601464 1080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.576596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.28.251.194:41777)
0808 03:57:28.592210 (+ 15614us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:28.592219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:28.592703 (+ 484us) server_negotiation.cc:407] Connection header received
0808 03:57:28.593031 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:28.593041 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:28.593327 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:28.593529 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:28.594431 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.595845 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.598362 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.599416 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.599616 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.599813 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:28.600953 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0808 03:57:28.601146 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15444,"thread_start_us":15300,"threads_started":1}
I20260808 03:57:28.602033 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.576137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41777 (local address 127.0.0.1:37494)
0808 03:57:28.592519 (+ 16382us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:28.592539 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:28.592814 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:28.593768 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:28.593772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:28.593798 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:28.594229 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.594242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.596015 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.596021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.598147 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.598160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.599648 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.599654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.600536 (+ 882us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.600569 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 03:57:28.600657 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 03:57:28.601292 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16266,"mutex_wait_us":261,"thread_start_us":16132,"threads_started":1}
I20260808 03:57:28.602142 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.575884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:35981 (local address 127.0.0.1:45588)
0808 03:57:28.590821 (+ 14937us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:28.590887 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:28.591092 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:28.592616 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:28.592624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:28.592678 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:28.593146 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.593159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.595306 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.595312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.597916 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.597932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.600097 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.600103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.600340 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.600412 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 03:57:28.601540 (+ 1128us) client_negotiation.cc:240] Negotiation successful
0808 03:57:28.601719 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14602,"mutex_wait_us":4,"thread_start_us":12697,"threads_started":1}
I20260808 03:57:28.605598 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.593054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:44511 (local address 127.0.0.1:33498)
0808 03:57:28.597426 (+ 4372us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:28.597441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:28.597570 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:28.598108 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:28.598112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:28.598144 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:28.598568 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.598579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.600628 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.600633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.602709 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.602725 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.604735 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.604742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.604925 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.604960 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 03:57:28.605066 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 03:57:28.605215 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4239,"mutex_wait_us":4,"thread_start_us":4081,"threads_started":1}
I20260808 03:57:28.605639 1083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.593139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.28.251.195:44511)
0808 03:57:28.596990 (+ 3851us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:28.597000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:28.597561 (+ 561us) server_negotiation.cc:407] Connection header received
0808 03:57:28.597713 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:28.597719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:28.597929 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:28.598123 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:28.598794 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.600468 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.602908 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.604483 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.604697 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.604904 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:28.605137 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 03:57:28.605310 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3676,"thread_start_us":3487,"threads_started":1}
I20260808 03:57:28.606238 685 tablet_service.cc:1467] Processing CreateTablet for tablet 5d942c4136cf4189b8ded2ef29882003 (DEFAULT_TABLE table=test-table [id=731e01c2d25a4184b1d0fd4e49374823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:28.606256 436 tablet_service.cc:1467] Processing CreateTablet for tablet 5d942c4136cf4189b8ded2ef29882003 (DEFAULT_TABLE table=test-table [id=731e01c2d25a4184b1d0fd4e49374823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:28.607779 436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d942c4136cf4189b8ded2ef29882003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:28.607788 685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d942c4136cf4189b8ded2ef29882003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:28.609935 926 tablet_service.cc:1467] Processing CreateTablet for tablet 5d942c4136cf4189b8ded2ef29882003 (DEFAULT_TABLE table=test-table [id=731e01c2d25a4184b1d0fd4e49374823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:28.611665 926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d942c4136cf4189b8ded2ef29882003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:28.620450 1086 tablet_bootstrap.cc:492] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c: Bootstrap starting.
I20260808 03:57:28.623894 1087 tablet_bootstrap.cc:492] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: Bootstrap starting.
I20260808 03:57:28.624094 1086 tablet_bootstrap.cc:654] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:28.625578 1086 log.cc:824] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:28.626588 1088 tablet_bootstrap.cc:492] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: Bootstrap starting.
I20260808 03:57:28.627719 1087 tablet_bootstrap.cc:654] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:28.629081 1087 log.cc:824] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:28.630223 1086 tablet_bootstrap.cc:492] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c: No bootstrap required, opened a new log
I20260808 03:57:28.630556 1086 ts_tablet_manager.cc:1397] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 03:57:28.630797 1088 tablet_bootstrap.cc:654] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:28.631951 1087 tablet_bootstrap.cc:492] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: No bootstrap required, opened a new log
I20260808 03:57:28.632138 1088 log.cc:824] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:28.632210 1087 ts_tablet_manager.cc:1397] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260808 03:57:28.635435 1088 tablet_bootstrap.cc:492] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: No bootstrap required, opened a new log
I20260808 03:57:28.635789 1088 ts_tablet_manager.cc:1397] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260808 03:57:28.639735 1087 raft_consensus.cc:348] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.640122 1087 raft_consensus.cc:374] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:28.640220 1087 raft_consensus.cc:729] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae8ed3767384f0eb0bd7f8d36890390, State: Initialized, Role: FOLLOWER
I20260808 03:57:28.640833 1087 consensus_queue.cc:260] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.641031 1086 raft_consensus.cc:348] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.641474 1086 raft_consensus.cc:374] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:28.641566 1086 raft_consensus.cc:729] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a21c0cfa099427890706cc2babfbe5c, State: Initialized, Role: FOLLOWER
I20260808 03:57:28.642185 1086 consensus_queue.cc:260] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.644508 810 heartbeater.cc:503] Master 127.28.251.254:43961 was elected leader, sending a full tablet report...
I20260808 03:57:28.645262 1087 ts_tablet_manager.cc:1428] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260808 03:57:28.646445 1086 ts_tablet_manager.cc:1428] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260808 03:57:28.647002 1088 raft_consensus.cc:348] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.647538 1088 raft_consensus.cc:374] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:28.647643 1088 raft_consensus.cc:729] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f52ac8f849f4a07b6555ad0ba250230, State: Initialized, Role: FOLLOWER
I20260808 03:57:28.648335 1088 consensus_queue.cc:260] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.649683 1042 heartbeater.cc:503] Master 127.28.251.254:43961 was elected leader, sending a full tablet report...
I20260808 03:57:28.650477 1088 ts_tablet_manager.cc:1428] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
W20260808 03:57:28.686573 811 tablet.cc:2369] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:28.730489 1094 raft_consensus.cc:482] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:28.730881 1094 raft_consensus.cc:504] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
W20260808 03:57:28.732281 1046 tablet.cc:2369] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:28.733106 1094 leader_election.cc:288] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981), 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195:44511)
I20260808 03:57:28.741036 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.733936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:45371 (local address 127.28.251.193:35981)
0808 03:57:28.734155 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:28.734162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:28.734979 (+ 817us) server_negotiation.cc:407] Connection header received
0808 03:57:28.735287 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:28.735294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:28.735374 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:28.735488 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:28.736276 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.737032 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.739148 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.739994 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.740138 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.740234 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:28.740642 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 03:57:28.740779 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 03:57:28.741050 1083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.734629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:57131 (local address 127.28.251.195:44511)
0808 03:57:28.734828 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:28.734834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:28.735284 (+ 450us) server_negotiation.cc:407] Connection header received
0808 03:57:28.735523 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:28.735529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:28.735616 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:28.735717 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:28.736384 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.737088 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.739127 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.740008 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.740173 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.740251 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:28.740641 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 03:57:28.740778 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 03:57:28.741196 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.733791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:44511 (local address 127.28.251.194:57131)
0808 03:57:28.735157 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:28.735176 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:28.735317 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:28.735786 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:28.735790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:28.735828 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:28.736215 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.736228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.737234 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.737242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.738954 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.738968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.740285 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.740290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.740443 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.740499 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 03:57:28.740583 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 03:57:28.740769 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"mutex_wait_us":52,"thread_start_us":627,"threads_started":1}
I20260808 03:57:28.741693 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.733365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:35981 (local address 127.28.251.194:45371)
0808 03:57:28.734854 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:28.734878 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:28.735102 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:28.735600 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:28.735605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:28.735647 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:28.736076 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.736088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.737174 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.737181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.738972 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.738986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.740243 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.740251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.740441 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.740495 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 03:57:28.740584 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 03:57:28.740772 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"mutex_wait_us":174,"thread_start_us":1492,"threads_started":1}
I20260808 03:57:28.742328 955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d942c4136cf4189b8ded2ef29882003" candidate_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f52ac8f849f4a07b6555ad0ba250230" is_pre_election: true
I20260808 03:57:28.743011 955 raft_consensus.cc:2454] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ae8ed3767384f0eb0bd7f8d36890390 in term 0.
I20260808 03:57:28.743005 469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d942c4136cf4189b8ded2ef29882003" candidate_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a21c0cfa099427890706cc2babfbe5c" is_pre_election: true
I20260808 03:57:28.743626 469 raft_consensus.cc:2454] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ae8ed3767384f0eb0bd7f8d36890390 in term 0.
I20260808 03:57:28.743935 592 leader_election.cc:302] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f52ac8f849f4a07b6555ad0ba250230, 3ae8ed3767384f0eb0bd7f8d36890390; no voters:
I20260808 03:57:28.744423 1094 raft_consensus.cc:2781] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:57:28.744586 1094 raft_consensus.cc:482] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:57:28.744776 1094 raft_consensus.cc:3037] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:28.747510 1094 raft_consensus.cc:504] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.748194 1094 leader_election.cc:288] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [CANDIDATE]: Term 1 election: Requested vote from peers 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981), 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195:44511)
I20260808 03:57:28.748762 469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d942c4136cf4189b8ded2ef29882003" candidate_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a21c0cfa099427890706cc2babfbe5c"
I20260808 03:57:28.748844 955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d942c4136cf4189b8ded2ef29882003" candidate_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f52ac8f849f4a07b6555ad0ba250230"
I20260808 03:57:28.748966 469 raft_consensus.cc:3037] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:28.749078 955 raft_consensus.cc:3037] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:28.751439 469 raft_consensus.cc:2454] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ae8ed3767384f0eb0bd7f8d36890390 in term 1.
I20260808 03:57:28.751713 955 raft_consensus.cc:2454] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ae8ed3767384f0eb0bd7f8d36890390 in term 1.
I20260808 03:57:28.752147 591 leader_election.cc:302] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a21c0cfa099427890706cc2babfbe5c, 3ae8ed3767384f0eb0bd7f8d36890390; no voters:
I20260808 03:57:28.752486 1094 raft_consensus.cc:2781] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:28.752967 1094 raft_consensus.cc:686] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 LEADER]: Becoming Leader. State: Replica: 3ae8ed3767384f0eb0bd7f8d36890390, State: Running, Role: LEADER
I20260808 03:57:28.753558 1094 consensus_queue.cc:237] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.758421 32660 catalog_manager.cc:5495] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194). New cstate: current_term: 1 leader_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } health_report { overall_health: UNKNOWN } } }
Aug 08 03:57:28 dist-test-slave-pww0 krb5kdc[32574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:28.820797 1120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.806310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43961 (local address 127.0.0.1:49924)
0808 03:57:28.807108 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:28.807126 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:28.807267 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:28.807608 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:28.807611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:28.808318 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:28.808602 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.808611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.809636 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.809639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.810630 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:28.810641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.811442 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.811446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:28.811589 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.812244 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:28.812266 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:28.814818 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:28.818155 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:28.818172 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:28.818179 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:28.818736 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:28.819129 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:28.819133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:28.819135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:28.819287 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:28.819859 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:28.819867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:28.820039 (+ 172us) client_negotiation.cc:769] Sending connection context
0808 03:57:28.820181 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 03:57:28.820447 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":384,"threads_started":1}
I20260808 03:57:28.820919 858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:28.806297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.28.251.254:43961)
0808 03:57:28.806592 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:28.806599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:28.807219 (+ 620us) server_negotiation.cc:407] Connection header received
0808 03:57:28.807369 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:28.807373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:28.807441 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:28.807572 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:28.808767 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.809515 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.810784 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:28.811235 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:28.811330 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:28.815072 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:28.815079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:28.815084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:28.815130 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:28.817968 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:28.818872 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:28.818876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:28.818878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:28.819010 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:28.819385 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:28.819389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:28.819391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:28.819733 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:28.819854 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:28.820286 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 03:57:28.820635 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":107}
I20260808 03:57:28.826397 32660 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:49924:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 03:57:28.827649 32660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:57:28.835050 685 tablet_service.cc:1467] Processing CreateTablet for tablet 57cc558adec74f44996085f986af586b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d1e6b71a59f47a6a461ff5dde629ed4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 03:57:28.834982 436 tablet_service.cc:1467] Processing CreateTablet for tablet 57cc558adec74f44996085f986af586b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d1e6b71a59f47a6a461ff5dde629ed4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 03:57:28.835275 926 tablet_service.cc:1467] Processing CreateTablet for tablet 57cc558adec74f44996085f986af586b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d1e6b71a59f47a6a461ff5dde629ed4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 03:57:28.836115 685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cc558adec74f44996085f986af586b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:28.836128 436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cc558adec74f44996085f986af586b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:28.836128 926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cc558adec74f44996085f986af586b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:28.842393 1087 tablet_bootstrap.cc:492] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390: Bootstrap starting.
I20260808 03:57:28.843672 1088 tablet_bootstrap.cc:492] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230: Bootstrap starting.
I20260808 03:57:28.843689 1086 tablet_bootstrap.cc:492] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: Bootstrap starting.
I20260808 03:57:28.844060 1087 tablet_bootstrap.cc:654] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:28.845435 1086 tablet_bootstrap.cc:654] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:28.845489 1088 tablet_bootstrap.cc:654] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:28.847301 1087 tablet_bootstrap.cc:492] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390: No bootstrap required, opened a new log
I20260808 03:57:28.847510 1087 ts_tablet_manager.cc:1397] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 03:57:28.847824 1086 tablet_bootstrap.cc:492] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: No bootstrap required, opened a new log
I20260808 03:57:28.847980 1086 ts_tablet_manager.cc:1397] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 03:57:28.848346 1087 raft_consensus.cc:348] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.848644 1088 tablet_bootstrap.cc:492] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230: No bootstrap required, opened a new log
I20260808 03:57:28.848625 1087 raft_consensus.cc:374] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:28.848737 1087 raft_consensus.cc:729] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae8ed3767384f0eb0bd7f8d36890390, State: Initialized, Role: FOLLOWER
I20260808 03:57:28.848620 1086 raft_consensus.cc:348] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.848798 1088 ts_tablet_manager.cc:1397] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260808 03:57:28.848866 1086 raft_consensus.cc:374] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:28.848966 1086 raft_consensus.cc:729] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a21c0cfa099427890706cc2babfbe5c, State: Initialized, Role: FOLLOWER
I20260808 03:57:28.849047 1087 consensus_queue.cc:260] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.849252 1086 consensus_queue.cc:260] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.849542 1094 tablet_replica.cc:439] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } } }
I20260808 03:57:28.849845 1094 tablet_replica.cc:442] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:28.849967 1086 ts_tablet_manager.cc:1428] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 03:57:28.850334 1087 ts_tablet_manager.cc:1428] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 03:57:28.850214 1095 tablet_replica.cc:439] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } } }
I20260808 03:57:28.850664 1095 tablet_replica.cc:442] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:28.850993 1088 raft_consensus.cc:348] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.851238 1088 raft_consensus.cc:374] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:28.851804 1088 raft_consensus.cc:729] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f52ac8f849f4a07b6555ad0ba250230, State: Initialized, Role: FOLLOWER
I20260808 03:57:28.852074 1088 consensus_queue.cc:260] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:28.852610 1097 tablet_replica.cc:439] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } } }
I20260808 03:57:28.853029 1097 tablet_replica.cc:442] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:28.853281 1088 ts_tablet_manager.cc:1428] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
W20260808 03:57:28.874598 558 tablet.cc:2369] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:29.112468 1095 raft_consensus.cc:482] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:29.112757 1095 raft_consensus.cc:504] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.114483 1095 leader_election.cc:288] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194:41777), 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195:44511)
I20260808 03:57:29.123445 1083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.115985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:47381 (local address 127.28.251.195:44511)
0808 03:57:29.116187 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:29.116192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:29.116470 (+ 278us) server_negotiation.cc:407] Connection header received
0808 03:57:29.116731 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:29.116736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:29.116803 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:29.116898 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:29.117416 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.118392 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.120946 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.122223 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.122398 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.122479 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:29.123003 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 03:57:29.123161 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 03:57:29.123696 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.115676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:44511 (local address 127.28.251.193:47381)
0808 03:57:29.116353 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:29.116370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:29.116527 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:29.116934 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:29.116937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:29.116959 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:29.117246 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.117254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.118541 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.118551 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.120724 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.120746 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.122525 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.122530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.122770 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.122821 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 03:57:29.122925 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 03:57:29.123127 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":93,"thread_start_us":310,"threads_started":1}
I20260808 03:57:29.123533 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.114925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:54787 (local address 127.28.251.194:41777)
0808 03:57:29.115934 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:29.115940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:29.115961 (+ 21us) server_negotiation.cc:407] Connection header received
0808 03:57:29.116042 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:29.116048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:29.116129 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:29.116246 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:29.116977 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.118104 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.120947 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.122219 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.122406 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.122479 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:29.123003 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 03:57:29.123161 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":694,"threads_started":1}
I20260808 03:57:29.123840 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.114771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41777 (local address 127.28.251.193:54787)
0808 03:57:29.115434 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:29.115478 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:29.115662 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:29.116289 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:29.116295 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:29.116329 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:29.116784 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.116795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.118257 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.118261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.120742 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.120751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.122508 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.122516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.122771 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.122822 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 03:57:29.122925 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 03:57:29.123127 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":146,"spinlock_wait_cycles":39296,"thread_start_us":355,"threads_started":1}
I20260808 03:57:29.124547 955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cc558adec74f44996085f986af586b" candidate_uuid: "0a21c0cfa099427890706cc2babfbe5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f52ac8f849f4a07b6555ad0ba250230" is_pre_election: true
I20260808 03:57:29.124874 955 raft_consensus.cc:2454] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a21c0cfa099427890706cc2babfbe5c in term 0.
I20260808 03:57:29.124861 717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cc558adec74f44996085f986af586b" candidate_uuid: "0a21c0cfa099427890706cc2babfbe5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" is_pre_election: true
I20260808 03:57:29.125331 717 raft_consensus.cc:2454] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a21c0cfa099427890706cc2babfbe5c in term 0.
I20260808 03:57:29.125564 351 leader_election.cc:302] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a21c0cfa099427890706cc2babfbe5c, 1f52ac8f849f4a07b6555ad0ba250230; no voters:
I20260808 03:57:29.126068 1095 raft_consensus.cc:2781] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:57:29.126217 1095 raft_consensus.cc:482] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:57:29.126329 1095 raft_consensus.cc:3037] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:29.128978 1095 raft_consensus.cc:504] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.129720 1095 leader_election.cc:288] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [CANDIDATE]: Term 1 election: Requested vote from peers 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194:41777), 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195:44511)
I20260808 03:57:29.130376 955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cc558adec74f44996085f986af586b" candidate_uuid: "0a21c0cfa099427890706cc2babfbe5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f52ac8f849f4a07b6555ad0ba250230"
I20260808 03:57:29.130426 717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cc558adec74f44996085f986af586b" candidate_uuid: "0a21c0cfa099427890706cc2babfbe5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ae8ed3767384f0eb0bd7f8d36890390"
I20260808 03:57:29.130635 717 raft_consensus.cc:3037] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:29.130672 955 raft_consensus.cc:3037] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:29.134068 717 raft_consensus.cc:2454] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a21c0cfa099427890706cc2babfbe5c in term 1.
I20260808 03:57:29.134253 955 raft_consensus.cc:2454] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a21c0cfa099427890706cc2babfbe5c in term 1.
I20260808 03:57:29.134600 351 leader_election.cc:302] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a21c0cfa099427890706cc2babfbe5c, 3ae8ed3767384f0eb0bd7f8d36890390; no voters:
I20260808 03:57:29.134922 1095 raft_consensus.cc:2781] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:29.135344 1095 raft_consensus.cc:686] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [term 1 LEADER]: Becoming Leader. State: Replica: 0a21c0cfa099427890706cc2babfbe5c, State: Running, Role: LEADER
I20260808 03:57:29.135855 1095 consensus_queue.cc:237] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.139189 1132 tablet_replica.cc:439] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: TxnStatusTablet state changed. Reason: New leader 0a21c0cfa099427890706cc2babfbe5c. Latest consensus state: current_term: 1 leader_uuid: "0a21c0cfa099427890706cc2babfbe5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } } }
I20260808 03:57:29.139573 1132 tablet_replica.cc:442] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: This TxnStatusTablet replica's current role is: LEADER
I20260808 03:57:29.140344 1136 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 03:57:29.140640 1136 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 03:57:29.141233 32660 catalog_manager.cc:5495] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193). New cstate: current_term: 1 leader_uuid: "0a21c0cfa099427890706cc2babfbe5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } health_report { overall_health: UNKNOWN } } }
I20260808 03:57:29.158179 1094 consensus_queue.cc:1035] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:29.166656 1094 consensus_queue.cc:1035] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:29.219233 1120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.213094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:35981 (local address 127.0.0.1:45604)
0808 03:57:29.213405 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:29.213428 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:29.213615 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:29.214135 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:29.214140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:29.214153 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 03:57:29.214357 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.214367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.215505 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.215511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.216571 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.216585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.217394 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.217400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.217579 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.217664 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.218630 (+ 966us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.218661 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 03:57:29.218740 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 03:57:29.218943 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 03:57:29.219251 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.213201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.28.251.193:35981)
0808 03:57:29.213423 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:29.213430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:29.213500 (+ 70us) server_negotiation.cc:407] Connection header received
0808 03:57:29.213850 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:29.213854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:29.213955 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:29.214093 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:29.214511 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.215323 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.216735 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.217204 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.217328 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.217873 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.218502 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.218660 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:29.218771 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 03:57:29.218865 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 03:57:29.227349 955 raft_consensus.cc:1261] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Refusing update from remote peer 0a21c0cfa099427890706cc2babfbe5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:29.227574 717 raft_consensus.cc:1261] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Refusing update from remote peer 0a21c0cfa099427890706cc2babfbe5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:29.228269 1132 consensus_queue.cc:1035] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:29.228683 1095 consensus_queue.cc:1035] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:29.236151 1094 tablet_replica.cc:439] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390: TxnStatusTablet state changed. Reason: New leader 0a21c0cfa099427890706cc2babfbe5c. Latest consensus state: current_term: 1 leader_uuid: "0a21c0cfa099427890706cc2babfbe5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } } }
I20260808 03:57:29.236545 1094 tablet_replica.cc:442] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:29.239413 1143 mvcc.cc:205] Tried to move back new op lower bound from 7316117296025174016 to 7316117295664332800. Current Snapshot: MvccSnapshot[applied={T|T < 7316117296025174016}]
I20260808 03:57:29.240476 1132 tablet_replica.cc:439] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a21c0cfa099427890706cc2babfbe5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } } }
I20260808 03:57:29.240823 1132 tablet_replica.cc:442] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: This TxnStatusTablet replica's current role is: LEADER
I20260808 03:57:29.246758 1094 tablet_replica.cc:439] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a21c0cfa099427890706cc2babfbe5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } } }
I20260808 03:57:29.247066 1094 tablet_replica.cc:442] T 57cc558adec74f44996085f986af586b P 3ae8ed3767384f0eb0bd7f8d36890390: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:29.247219 1097 tablet_replica.cc:439] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230: TxnStatusTablet state changed. Reason: New leader 0a21c0cfa099427890706cc2babfbe5c. Latest consensus state: current_term: 1 leader_uuid: "0a21c0cfa099427890706cc2babfbe5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } } }
I20260808 03:57:29.247599 1097 tablet_replica.cc:442] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:29.250492 1132 tablet_replica.cc:439] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a21c0cfa099427890706cc2babfbe5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } } }
I20260808 03:57:29.250808 1132 tablet_replica.cc:442] T 57cc558adec74f44996085f986af586b P 0a21c0cfa099427890706cc2babfbe5c: This TxnStatusTablet replica's current role is: LEADER
I20260808 03:57:29.258572 1097 tablet_replica.cc:439] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a21c0cfa099427890706cc2babfbe5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } } }
I20260808 03:57:29.258872 1097 tablet_replica.cc:442] T 57cc558adec74f44996085f986af586b P 1f52ac8f849f4a07b6555ad0ba250230: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:57:29.267163 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.257675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.28.251.194:41777)
0808 03:57:29.257882 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:29.257889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:29.260164 (+ 2275us) server_negotiation.cc:407] Connection header received
0808 03:57:29.260600 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:29.260607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:29.260733 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:29.260871 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:29.261626 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.262690 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.264780 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.265302 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.265421 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.265959 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.266560 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.266670 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:29.266799 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 03:57:29.266892 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 03:57:29.285389 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.278803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:35981 (local address 127.28.251.194:58261)
0808 03:57:29.279446 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:29.279486 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:29.279653 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:29.280027 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:29.280030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:29.280039 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 03:57:29.280277 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.280285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.281765 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.281769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.282944 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.282959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.284173 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.284179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.284310 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.284382 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.284877 (+ 495us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.284894 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 03:57:29.284955 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 03:57:29.285074 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"spinlock_wait_cycles":1792,"thread_start_us":441,"threads_started":1}
I20260808 03:57:29.285411 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.278995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:58261 (local address 127.28.251.193:35981)
0808 03:57:29.279238 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:29.279244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:29.279587 (+ 343us) server_negotiation.cc:407] Connection header received
0808 03:57:29.279796 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:29.279802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:29.279902 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:29.280002 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:29.280431 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.281594 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.283102 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.283984 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.284131 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.284623 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.284773 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.284873 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:29.285006 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 03:57:29.285100 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 03:57:29.330300 1168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.322558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41777 (local address 127.28.251.193:45787)
0808 03:57:29.323305 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:29.323325 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:29.323516 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:29.323929 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:29.323934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:29.323947 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 03:57:29.324391 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.324404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.325995 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.326002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.327933 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.327958 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.328953 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.328960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.329165 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.329254 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.329757 (+ 503us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.329769 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 03:57:29.329849 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 03:57:29.329978 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":366,"threads_started":1}
I20260808 03:57:29.330305 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.322556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:45787 (local address 127.28.251.194:41777)
0808 03:57:29.322771 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:29.322777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:29.323425 (+ 648us) server_negotiation.cc:407] Connection header received
0808 03:57:29.323677 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:29.323683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:29.323790 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:29.323888 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:29.324572 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.325850 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.328121 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.328741 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.328887 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.329487 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.329642 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.329739 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:29.329893 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 03:57:29.329989 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 03:57:29.374666 32667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:29.374982 32667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:29.379549 32667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc: Sending DeleteTablet for 3 replicas of tablet 5d942c4136cf4189b8ded2ef29882003
I20260808 03:57:29.380828 685 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d942c4136cf4189b8ded2ef29882003 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:29 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:37494
I20260808 03:57:29.380848 436 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d942c4136cf4189b8ded2ef29882003 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:29 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:45588
I20260808 03:57:29.382107 926 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d942c4136cf4189b8ded2ef29882003 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:29 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:33498
I20260808 03:57:29.390622 32660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:57:29.391510 32660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:57:29.392699 1177 tablet_replica.cc:330] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c: stopping tablet replica
I20260808 03:57:29.393410 1177 raft_consensus.cc:2229] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:29.399098 1177 raft_consensus.cc:2258] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:29.401688 1177 ts_tablet_manager.cc:1905] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:29.402740 436 tablet_service.cc:1467] Processing CreateTablet for tablet 89521591f42a4b8d8ec94fa8e01f1069 (DEFAULT_TABLE table=test-table [id=f1b17267f2e640939da0b7c2ac109bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:29.403014 681 tablet_service.cc:1467] Processing CreateTablet for tablet 89521591f42a4b8d8ec94fa8e01f1069 (DEFAULT_TABLE table=test-table [id=f1b17267f2e640939da0b7c2ac109bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:29.403685 681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89521591f42a4b8d8ec94fa8e01f1069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:29.403976 436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89521591f42a4b8d8ec94fa8e01f1069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:29.408196 924 tablet_service.cc:1467] Processing CreateTablet for tablet 89521591f42a4b8d8ec94fa8e01f1069 (DEFAULT_TABLE table=test-table [id=f1b17267f2e640939da0b7c2ac109bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:29.408728 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89521591f42a4b8d8ec94fa8e01f1069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:29.421509 1179 tablet_replica.cc:330] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: stopping tablet replica
I20260808 03:57:29.422019 1179 raft_consensus.cc:2229] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:29.422452 1179 raft_consensus.cc:2258] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:29.423138 1180 tablet_bootstrap.cc:492] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c: Bootstrap starting.
I20260808 03:57:29.425583 1180 tablet_bootstrap.cc:654] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:29.426638 1176 tablet_replica.cc:330] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: stopping tablet replica
I20260808 03:57:29.425743 1177 ts_tablet_manager.cc:1918] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:57:29.427009 1177 log.cc:1196] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/5d942c4136cf4189b8ded2ef29882003
I20260808 03:57:29.427171 1176 raft_consensus.cc:2229] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:57:29.428046 1177 ts_tablet_manager.cc:1939] T 5d942c4136cf4189b8ded2ef29882003 P 0a21c0cfa099427890706cc2babfbe5c: Deleting consensus metadata
I20260808 03:57:29.429749 1179 ts_tablet_manager.cc:1905] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:29.430747 32636 catalog_manager.cc:4841] TS 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981): tablet 5d942c4136cf4189b8ded2ef29882003 (table test-table [id=731e01c2d25a4184b1d0fd4e49374823]) successfully deleted
I20260808 03:57:29.435325 1183 tablet_bootstrap.cc:492] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230: Bootstrap starting.
I20260808 03:57:29.437530 1183 tablet_bootstrap.cc:654] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:29.441128 1176 raft_consensus.cc:2258] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:29.441351 1180 tablet_bootstrap.cc:492] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c: No bootstrap required, opened a new log
I20260808 03:57:29.442072 1180 ts_tablet_manager.cc:1397] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20260808 03:57:29.443288 1180 raft_consensus.cc:348] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.444896 1180 raft_consensus.cc:374] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:29.445130 1180 raft_consensus.cc:729] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a21c0cfa099427890706cc2babfbe5c, State: Initialized, Role: FOLLOWER
I20260808 03:57:29.445997 1176 ts_tablet_manager.cc:1905] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:29.451074 1181 tablet_bootstrap.cc:492] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390: Bootstrap starting.
I20260808 03:57:29.451450 1180 consensus_queue.cc:260] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.453868 1183 tablet_bootstrap.cc:492] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230: No bootstrap required, opened a new log
I20260808 03:57:29.454114 1183 ts_tablet_manager.cc:1397] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260808 03:57:29.455128 1183 raft_consensus.cc:348] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.455436 1183 raft_consensus.cc:374] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:29.455585 1183 raft_consensus.cc:729] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f52ac8f849f4a07b6555ad0ba250230, State: Initialized, Role: FOLLOWER
I20260808 03:57:29.455943 1183 consensus_queue.cc:260] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.457405 1179 ts_tablet_manager.cc:1918] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:57:29.457597 1179 log.cc:1196] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/5d942c4136cf4189b8ded2ef29882003
I20260808 03:57:29.458244 1179 ts_tablet_manager.cc:1939] T 5d942c4136cf4189b8ded2ef29882003 P 1f52ac8f849f4a07b6555ad0ba250230: Deleting consensus metadata
I20260808 03:57:29.459055 1183 ts_tablet_manager.cc:1428] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260808 03:57:29.459352 1180 ts_tablet_manager.cc:1428] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20260808 03:57:29.460542 32638 catalog_manager.cc:4841] TS 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195:44511): tablet 5d942c4136cf4189b8ded2ef29882003 (table test-table [id=731e01c2d25a4184b1d0fd4e49374823]) successfully deleted
I20260808 03:57:29.463819 1181 tablet_bootstrap.cc:654] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:29.469089 1176 ts_tablet_manager.cc:1918] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:57:29.469398 1176 log.cc:1196] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/5d942c4136cf4189b8ded2ef29882003
I20260808 03:57:29.469910 1181 tablet_bootstrap.cc:492] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390: No bootstrap required, opened a new log
I20260808 03:57:29.470094 1181 ts_tablet_manager.cc:1397] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260808 03:57:29.470235 1176 ts_tablet_manager.cc:1939] T 5d942c4136cf4189b8ded2ef29882003 P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting consensus metadata
I20260808 03:57:29.471027 1181 raft_consensus.cc:348] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.471302 1181 raft_consensus.cc:374] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:29.471429 1181 raft_consensus.cc:729] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae8ed3767384f0eb0bd7f8d36890390, State: Initialized, Role: FOLLOWER
I20260808 03:57:29.471779 1181 consensus_queue.cc:260] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.472704 32638 catalog_manager.cc:4841] TS 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194:41777): tablet 5d942c4136cf4189b8ded2ef29882003 (table test-table [id=731e01c2d25a4184b1d0fd4e49374823]) successfully deleted
I20260808 03:57:29.473109 1181 ts_tablet_manager.cc:1428] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:29.481287 1097 raft_consensus.cc:482] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:29.481556 1097 raft_consensus.cc:504] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.483630 1097 leader_election.cc:288] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981), 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194:41777)
I20260808 03:57:29.524403 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.486533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:34073 (local address 127.28.251.194:41777)
0808 03:57:29.486730 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:29.486736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:29.511876 (+ 25140us) server_negotiation.cc:407] Connection header received
0808 03:57:29.513234 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:29.513240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:29.513338 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:29.513405 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:29.515096 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.516243 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.520533 (+ 4290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.521977 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.522161 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.522266 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:29.523892 (+ 1626us) server_negotiation.cc:299] Negotiation successful
0808 03:57:29.524070 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 03:57:29.525461 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.486026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41777 (local address 127.28.251.195:34073)
0808 03:57:29.511736 (+ 25710us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:29.511764 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:29.512617 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:29.513973 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:29.513978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:29.514017 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:29.514381 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.514393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.516452 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.516458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.518849 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.518868 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.522499 (+ 3631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.522506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.523623 (+ 1117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.523678 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 03:57:29.524904 (+ 1226us) client_negotiation.cc:240] Negotiation successful
0808 03:57:29.525082 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25553,"thread_start_us":25201,"threads_started":1}
I20260808 03:57:29.525674 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.485320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:33985 (local address 127.28.251.193:35981)
0808 03:57:29.485547 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:29.485553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:29.512240 (+ 26687us) server_negotiation.cc:407] Connection header received
0808 03:57:29.512853 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:29.512861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:29.512946 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:29.513011 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:29.519095 (+ 6084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.520302 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.522654 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.524168 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.524401 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.524483 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:29.525181 (+ 698us) server_negotiation.cc:299] Negotiation successful
0808 03:57:29.525350 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 03:57:29.526574 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.484480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:35981 (local address 127.28.251.195:33985)
0808 03:57:29.512145 (+ 27665us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:29.512163 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:29.512460 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:29.513639 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:29.513643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:29.513673 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:29.514865 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.514879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.520432 (+ 5553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.520438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.522458 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.522482 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.524761 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.524768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.524976 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.525014 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 03:57:29.526047 (+ 1033us) client_negotiation.cc:240] Negotiation successful
0808 03:57:29.526207 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26705,"thread_start_us":35598,"threads_started":1}
I20260808 03:57:29.526619 717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89521591f42a4b8d8ec94fa8e01f1069" candidate_uuid: "1f52ac8f849f4a07b6555ad0ba250230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" is_pre_election: true
I20260808 03:57:29.526921 717 raft_consensus.cc:2454] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f52ac8f849f4a07b6555ad0ba250230 in term 0.
I20260808 03:57:29.527746 469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89521591f42a4b8d8ec94fa8e01f1069" candidate_uuid: "1f52ac8f849f4a07b6555ad0ba250230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a21c0cfa099427890706cc2babfbe5c" is_pre_election: true
I20260808 03:57:29.527727 844 leader_election.cc:302] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f52ac8f849f4a07b6555ad0ba250230, 3ae8ed3767384f0eb0bd7f8d36890390; no voters:
I20260808 03:57:29.528067 469 raft_consensus.cc:2454] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f52ac8f849f4a07b6555ad0ba250230 in term 0.
I20260808 03:57:29.528510 1097 raft_consensus.cc:2781] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:57:29.528653 1097 raft_consensus.cc:482] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:57:29.528766 1097 raft_consensus.cc:3037] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:29.532230 1097 raft_consensus.cc:504] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.533286 1097 leader_election.cc:288] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [CANDIDATE]: Term 1 election: Requested vote from peers 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981), 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194:41777)
I20260808 03:57:29.533931 469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89521591f42a4b8d8ec94fa8e01f1069" candidate_uuid: "1f52ac8f849f4a07b6555ad0ba250230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a21c0cfa099427890706cc2babfbe5c"
I20260808 03:57:29.534210 469 raft_consensus.cc:3037] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:29.534322 717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89521591f42a4b8d8ec94fa8e01f1069" candidate_uuid: "1f52ac8f849f4a07b6555ad0ba250230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ae8ed3767384f0eb0bd7f8d36890390"
I20260808 03:57:29.534591 717 raft_consensus.cc:3037] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:29.537768 469 raft_consensus.cc:2454] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f52ac8f849f4a07b6555ad0ba250230 in term 1.
I20260808 03:57:29.537920 717 raft_consensus.cc:2454] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f52ac8f849f4a07b6555ad0ba250230 in term 1.
I20260808 03:57:29.538647 843 leader_election.cc:302] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a21c0cfa099427890706cc2babfbe5c, 1f52ac8f849f4a07b6555ad0ba250230; no voters:
I20260808 03:57:29.539108 1097 raft_consensus.cc:2781] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:29.539696 1097 raft_consensus.cc:686] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 LEADER]: Becoming Leader. State: Replica: 1f52ac8f849f4a07b6555ad0ba250230, State: Running, Role: LEADER
I20260808 03:57:29.540194 1097 consensus_queue.cc:237] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:29.544620 32660 catalog_manager.cc:5495] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195). New cstate: current_term: 1 leader_uuid: "1f52ac8f849f4a07b6555ad0ba250230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } health_report { overall_health: HEALTHY } } }
I20260808 03:57:29.585953 1083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.578817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.28.251.195:44511)
0808 03:57:29.579008 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:29.579018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:29.579037 (+ 19us) server_negotiation.cc:407] Connection header received
0808 03:57:29.579152 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:29.579157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:29.579271 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:29.579495 (+ 224us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:29.579923 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.581053 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.583032 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.583718 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.583857 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.584454 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.585263 (+ 809us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.585414 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:29.585525 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 03:57:29.585621 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 03:57:29.606858 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.599113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:35981 (local address 127.28.251.195:51185)
0808 03:57:29.599860 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:29.599881 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:29.600108 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:29.600506 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:29.600511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:29.600524 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 03:57:29.600871 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.600882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.602365 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.602370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.604141 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.604160 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.605249 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.605256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.605459 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.605557 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.606156 (+ 599us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.606166 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 03:57:29.606248 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 03:57:29.606415 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":702,"threads_started":1}
I20260808 03:57:29.606856 1078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.599626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:51185 (local address 127.28.251.193:35981)
0808 03:57:29.599820 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:29.599826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:29.599975 (+ 149us) server_negotiation.cc:407] Connection header received
0808 03:57:29.600280 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:29.600286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:29.600379 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:29.600480 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:29.601035 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.602210 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.604316 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.605025 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.605183 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.605807 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.605992 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.606066 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:29.606320 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 03:57:29.606433 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 03:57:29.619758 717 raft_consensus.cc:1261] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Refusing update from remote peer 1f52ac8f849f4a07b6555ad0ba250230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:29.620160 469 raft_consensus.cc:1261] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Refusing update from remote peer 1f52ac8f849f4a07b6555ad0ba250230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:29.620714 1197 consensus_queue.cc:1035] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:29.621258 1097 consensus_queue.cc:1035] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:29.640653 1146 mvcc.cc:205] Tried to move back new op lower bound from 7316117297635672064 to 7316117297321918464. Current Snapshot: MvccSnapshot[applied={T|T < 7316117297635672064}]
I20260808 03:57:29.675161 1168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.668985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:44511 (local address 127.28.251.193:37857)
0808 03:57:29.669853 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:29.669875 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:29.670002 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:29.670382 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:29.670387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:29.670400 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 03:57:29.670583 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.670592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.671943 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.671948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.672957 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:29.672971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.673849 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.673855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:29.674029 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.674085 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.674603 (+ 518us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.674615 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 03:57:29.674718 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 03:57:29.674867 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718}
I20260808 03:57:29.675177 1083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:29.669096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:37857 (local address 127.28.251.195:44511)
0808 03:57:29.669249 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:29.669253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:29.669959 (+ 706us) server_negotiation.cc:407] Connection header received
0808 03:57:29.670150 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:29.670155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:29.670248 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:29.670330 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:29.670857 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.671805 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.673102 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:29.673694 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:29.673813 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:29.674272 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:29.674430 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:29.674523 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:29.674769 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 03:57:29.674856 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 03:57:29.676415 1146 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 03:57:31.330411 32660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:31.330729 32660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:31.336062 32660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc: Sending DeleteTablet for 3 replicas of tablet 89521591f42a4b8d8ec94fa8e01f1069
I20260808 03:57:31.337020 436 tablet_service.cc:1514] Processing DeleteTablet for tablet 89521591f42a4b8d8ec94fa8e01f1069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:31 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:45588
I20260808 03:57:31.337047 681 tablet_service.cc:1514] Processing DeleteTablet for tablet 89521591f42a4b8d8ec94fa8e01f1069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:31 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:37494
I20260808 03:57:31.337745 1258 tablet_replica.cc:330] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c: stopping tablet replica
I20260808 03:57:31.338035 1258 raft_consensus.cc:2229] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:31.338255 924 tablet_service.cc:1514] Processing DeleteTablet for tablet 89521591f42a4b8d8ec94fa8e01f1069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:31 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:33498
I20260808 03:57:31.338495 1258 raft_consensus.cc:2258] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:31.339823 1258 ts_tablet_manager.cc:1905] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:31.347146 1258 ts_tablet_manager.cc:1918] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 03:57:31.347424 1258 log.cc:1196] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/89521591f42a4b8d8ec94fa8e01f1069
I20260808 03:57:31.348012 1258 ts_tablet_manager.cc:1939] T 89521591f42a4b8d8ec94fa8e01f1069 P 0a21c0cfa099427890706cc2babfbe5c: Deleting consensus metadata
I20260808 03:57:31.349350 32636 catalog_manager.cc:4841] TS 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981): tablet 89521591f42a4b8d8ec94fa8e01f1069 (table test-table [id=f1b17267f2e640939da0b7c2ac109bee]) successfully deleted
I20260808 03:57:31.352993 1261 tablet_replica.cc:330] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390: stopping tablet replica
I20260808 03:57:31.353242 1261 raft_consensus.cc:2229] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:31.353533 1261 raft_consensus.cc:2258] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:31.354758 1261 ts_tablet_manager.cc:1905] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:31.361318 1263 tablet_replica.cc:330] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230: stopping tablet replica
I20260808 03:57:31.361545 1263 raft_consensus.cc:2229] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:57:31.362191 1261 ts_tablet_manager.cc:1918] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 03:57:31.362401 1261 log.cc:1196] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/89521591f42a4b8d8ec94fa8e01f1069
I20260808 03:57:31.362882 1261 ts_tablet_manager.cc:1939] T 89521591f42a4b8d8ec94fa8e01f1069 P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting consensus metadata
I20260808 03:57:31.364341 32638 catalog_manager.cc:4841] TS 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194:41777): tablet 89521591f42a4b8d8ec94fa8e01f1069 (table test-table [id=f1b17267f2e640939da0b7c2ac109bee]) successfully deleted
I20260808 03:57:31.369499 1263 raft_consensus.cc:2258] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:31.371093 1263 ts_tablet_manager.cc:1905] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:31.377398 1263 ts_tablet_manager.cc:1918] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 03:57:31.377576 1263 log.cc:1196] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/89521591f42a4b8d8ec94fa8e01f1069
I20260808 03:57:31.378134 1263 ts_tablet_manager.cc:1939] T 89521591f42a4b8d8ec94fa8e01f1069 P 1f52ac8f849f4a07b6555ad0ba250230: Deleting consensus metadata
I20260808 03:57:31.379767 32638 catalog_manager.cc:4841] TS 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195:44511): tablet 89521591f42a4b8d8ec94fa8e01f1069 (table test-table [id=f1b17267f2e640939da0b7c2ac109bee]) successfully deleted
Aug 08 03:57:31 dist-test-slave-pww0 krb5kdc[32574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 03:57:31 dist-test-slave-pww0 krb5kdc[32574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:31.426343 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:31.410645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.28.251.254:43961)
0808 03:57:31.411433 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:31.411438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:31.411485 (+ 47us) server_negotiation.cc:407] Connection header received
0808 03:57:31.411552 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:31.411556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:31.411630 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:31.411728 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:31.413157 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:31.414110 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:31.415520 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:31.415958 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:31.416078 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:31.420434 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:31.420442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:31.420448 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:31.420501 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:31.423687 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:31.424382 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:31.424387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:31.424389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:31.424523 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:31.424906 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:31.424910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:31.424912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:31.425195 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:31.425429 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:31.425760 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 03:57:31.425996 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":466,"threads_started":1}
I20260808 03:57:31.430023 32660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:57:31.430656 32660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:57:31.437770 436 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5be254dfea4a3590c9cd5d514a4f91 (DEFAULT_TABLE table=test-table [id=fd25e2c0c73d400c8b19064c02de89f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:31.437924 681 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5be254dfea4a3590c9cd5d514a4f91 (DEFAULT_TABLE table=test-table [id=fd25e2c0c73d400c8b19064c02de89f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:31.438309 436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5be254dfea4a3590c9cd5d514a4f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:31.438408 681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5be254dfea4a3590c9cd5d514a4f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:31.438324 924 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5be254dfea4a3590c9cd5d514a4f91 (DEFAULT_TABLE table=test-table [id=fd25e2c0c73d400c8b19064c02de89f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:31.438874 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5be254dfea4a3590c9cd5d514a4f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:31.450829 1274 tablet_bootstrap.cc:492] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390: Bootstrap starting.
I20260808 03:57:31.452857 1274 tablet_bootstrap.cc:654] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:31.455183 1273 tablet_bootstrap.cc:492] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230: Bootstrap starting.
I20260808 03:57:31.457491 1273 tablet_bootstrap.cc:654] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:31.459661 1275 tablet_bootstrap.cc:492] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c: Bootstrap starting.
I20260808 03:57:31.460377 1274 tablet_bootstrap.cc:492] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390: No bootstrap required, opened a new log
I20260808 03:57:31.460548 1274 ts_tablet_manager.cc:1397] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260808 03:57:31.461396 1274 raft_consensus.cc:348] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.461606 1274 raft_consensus.cc:374] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:31.461620 1275 tablet_bootstrap.cc:654] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:31.461696 1274 raft_consensus.cc:729] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae8ed3767384f0eb0bd7f8d36890390, State: Initialized, Role: FOLLOWER
I20260808 03:57:31.461942 1274 consensus_queue.cc:260] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.463198 1274 ts_tablet_manager.cc:1428] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:31.465549 1273 tablet_bootstrap.cc:492] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230: No bootstrap required, opened a new log
I20260808 03:57:31.465768 1273 ts_tablet_manager.cc:1397] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260808 03:57:31.466143 1275 tablet_bootstrap.cc:492] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c: No bootstrap required, opened a new log
I20260808 03:57:31.466322 1275 ts_tablet_manager.cc:1397] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260808 03:57:31.466835 1273 raft_consensus.cc:348] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.467134 1273 raft_consensus.cc:374] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:31.467271 1273 raft_consensus.cc:729] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f52ac8f849f4a07b6555ad0ba250230, State: Initialized, Role: FOLLOWER
I20260808 03:57:31.467167 1275 raft_consensus.cc:348] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.467487 1275 raft_consensus.cc:374] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:31.467681 1275 raft_consensus.cc:729] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a21c0cfa099427890706cc2babfbe5c, State: Initialized, Role: FOLLOWER
I20260808 03:57:31.467669 1273 consensus_queue.cc:260] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.467980 1275 consensus_queue.cc:260] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.468315 1273 ts_tablet_manager.cc:1428] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 03:57:31.468587 1275 ts_tablet_manager.cc:1428] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 03:57:31.580564 1251 raft_consensus.cc:482] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:31.580833 1251 raft_consensus.cc:504] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.581650 1251 leader_election.cc:288] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981), 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194:41777)
I20260808 03:57:31.582314 468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5be254dfea4a3590c9cd5d514a4f91" candidate_uuid: "1f52ac8f849f4a07b6555ad0ba250230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a21c0cfa099427890706cc2babfbe5c" is_pre_election: true
I20260808 03:57:31.582324 717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5be254dfea4a3590c9cd5d514a4f91" candidate_uuid: "1f52ac8f849f4a07b6555ad0ba250230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" is_pre_election: true
I20260808 03:57:31.582569 717 raft_consensus.cc:2454] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f52ac8f849f4a07b6555ad0ba250230 in term 0.
I20260808 03:57:31.582561 468 raft_consensus.cc:2454] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f52ac8f849f4a07b6555ad0ba250230 in term 0.
I20260808 03:57:31.583073 844 leader_election.cc:302] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a21c0cfa099427890706cc2babfbe5c, 1f52ac8f849f4a07b6555ad0ba250230, 3ae8ed3767384f0eb0bd7f8d36890390; no voters:
I20260808 03:57:31.583390 1251 raft_consensus.cc:2781] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:57:31.583544 1251 raft_consensus.cc:482] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:57:31.583648 1251 raft_consensus.cc:3037] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:31.585824 1251 raft_consensus.cc:504] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.586447 1251 leader_election.cc:288] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [CANDIDATE]: Term 1 election: Requested vote from peers 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981), 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194:41777)
I20260808 03:57:31.587005 468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5be254dfea4a3590c9cd5d514a4f91" candidate_uuid: "1f52ac8f849f4a07b6555ad0ba250230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a21c0cfa099427890706cc2babfbe5c"
I20260808 03:57:31.587011 717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5be254dfea4a3590c9cd5d514a4f91" candidate_uuid: "1f52ac8f849f4a07b6555ad0ba250230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ae8ed3767384f0eb0bd7f8d36890390"
I20260808 03:57:31.587230 468 raft_consensus.cc:3037] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:31.587220 717 raft_consensus.cc:3037] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:31.589469 717 raft_consensus.cc:2454] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f52ac8f849f4a07b6555ad0ba250230 in term 1.
I20260808 03:57:31.589476 468 raft_consensus.cc:2454] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f52ac8f849f4a07b6555ad0ba250230 in term 1.
I20260808 03:57:31.589910 844 leader_election.cc:302] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f52ac8f849f4a07b6555ad0ba250230, 3ae8ed3767384f0eb0bd7f8d36890390; no voters:
I20260808 03:57:31.590216 1251 raft_consensus.cc:2781] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:31.590386 1251 raft_consensus.cc:686] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 LEADER]: Becoming Leader. State: Replica: 1f52ac8f849f4a07b6555ad0ba250230, State: Running, Role: LEADER
I20260808 03:57:31.590698 1251 consensus_queue.cc:237] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.593688 32660 catalog_manager.cc:5495] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195). New cstate: current_term: 1 leader_uuid: "1f52ac8f849f4a07b6555ad0ba250230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } health_report { overall_health: HEALTHY } } }
I20260808 03:57:31.622184 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:31.616726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.28.251.195:44511)
0808 03:57:31.617331 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:31.617336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:31.617360 (+ 24us) server_negotiation.cc:407] Connection header received
0808 03:57:31.617427 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:31.617431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:31.617501 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:31.617606 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:31.617967 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:31.618812 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:31.620216 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:31.620676 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:31.620779 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:31.621392 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:31.621509 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:31.621580 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:31.621744 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 03:57:31.621817 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":308,"threads_started":1}
I20260808 03:57:31.632588 717 raft_consensus.cc:1261] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Refusing update from remote peer 1f52ac8f849f4a07b6555ad0ba250230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:31.632648 468 raft_consensus.cc:1261] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Refusing update from remote peer 1f52ac8f849f4a07b6555ad0ba250230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:31.633221 1251 consensus_queue.cc:1035] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:31.633504 1247 consensus_queue.cc:1035] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:31.638965 1217 mvcc.cc:205] Tried to move back new op lower bound from 7316117305881559040 to 7316117305717866496. Current Snapshot: MvccSnapshot[applied={T|T < 7316117305881559040}]
I20260808 03:57:31.679059 32665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:31.679298 32665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:31.682328 32665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc: Sending DeleteTablet for 3 replicas of tablet 1a5be254dfea4a3590c9cd5d514a4f91
I20260808 03:57:31.683260 681 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5be254dfea4a3590c9cd5d514a4f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:31 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:37494
I20260808 03:57:31.683257 436 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5be254dfea4a3590c9cd5d514a4f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:31 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:45588
I20260808 03:57:31.683631 1258 tablet_replica.cc:330] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c: stopping tablet replica
I20260808 03:57:31.683634 924 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5be254dfea4a3590c9cd5d514a4f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:31 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:33498
I20260808 03:57:31.683876 1258 raft_consensus.cc:2229] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:31.683959 1263 tablet_replica.cc:330] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230: stopping tablet replica
I20260808 03:57:31.683993 1261 tablet_replica.cc:330] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390: stopping tablet replica
I20260808 03:57:31.684103 1258 raft_consensus.cc:2258] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:31.684194 1261 raft_consensus.cc:2229] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:31.684193 1263 raft_consensus.cc:2229] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:57:31.684391 1261 raft_consensus.cc:2258] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:31.684610 1263 raft_consensus.cc:2258] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:31.684876 32665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 03:57:31.685480 1258 ts_tablet_manager.cc:1905] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 03:57:31.685650 32665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:57:31.685819 1263 ts_tablet_manager.cc:1905] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:31.686455 1261 ts_tablet_manager.cc:1905] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:31.692482 1258 ts_tablet_manager.cc:1918] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:57:31.692718 1258 log.cc:1196] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/1a5be254dfea4a3590c9cd5d514a4f91
I20260808 03:57:31.693151 1263 ts_tablet_manager.cc:1918] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:57:31.693218 1258 ts_tablet_manager.cc:1939] T 1a5be254dfea4a3590c9cd5d514a4f91 P 0a21c0cfa099427890706cc2babfbe5c: Deleting consensus metadata
I20260808 03:57:31.693341 1263 log.cc:1196] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/1a5be254dfea4a3590c9cd5d514a4f91
I20260808 03:57:31.694205 436 tablet_service.cc:1467] Processing CreateTablet for tablet f5ca74423eef4206af7cad37ecb8243d (DEFAULT_TABLE table=test-table [id=410bc99c58314d5aaa3fa43786685733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:31.694268 924 tablet_service.cc:1467] Processing CreateTablet for tablet f5ca74423eef4206af7cad37ecb8243d (DEFAULT_TABLE table=test-table [id=410bc99c58314d5aaa3fa43786685733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:31.694301 681 tablet_service.cc:1467] Processing CreateTablet for tablet f5ca74423eef4206af7cad37ecb8243d (DEFAULT_TABLE table=test-table [id=410bc99c58314d5aaa3fa43786685733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:31.694674 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ca74423eef4206af7cad37ecb8243d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:31.694697 436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ca74423eef4206af7cad37ecb8243d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:31.694700 681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ca74423eef4206af7cad37ecb8243d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:31.695823 1263 ts_tablet_manager.cc:1939] T 1a5be254dfea4a3590c9cd5d514a4f91 P 1f52ac8f849f4a07b6555ad0ba250230: Deleting consensus metadata
I20260808 03:57:31.699774 32636 catalog_manager.cc:4841] TS 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981): tablet 1a5be254dfea4a3590c9cd5d514a4f91 (table test-table [id=fd25e2c0c73d400c8b19064c02de89f7]) successfully deleted
I20260808 03:57:31.700299 1274 tablet_bootstrap.cc:492] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390: Bootstrap starting.
I20260808 03:57:31.700855 1261 ts_tablet_manager.cc:1918] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:57:31.700955 32638 catalog_manager.cc:4841] TS 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195:44511): tablet 1a5be254dfea4a3590c9cd5d514a4f91 (table test-table [id=fd25e2c0c73d400c8b19064c02de89f7]) successfully deleted
I20260808 03:57:31.701071 1261 log.cc:1196] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/1a5be254dfea4a3590c9cd5d514a4f91
I20260808 03:57:31.701678 1261 ts_tablet_manager.cc:1939] T 1a5be254dfea4a3590c9cd5d514a4f91 P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting consensus metadata
I20260808 03:57:31.702116 1274 tablet_bootstrap.cc:654] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:31.703230 32638 catalog_manager.cc:4841] TS 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194:41777): tablet 1a5be254dfea4a3590c9cd5d514a4f91 (table test-table [id=fd25e2c0c73d400c8b19064c02de89f7]) successfully deleted
I20260808 03:57:31.703775 1273 tablet_bootstrap.cc:492] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230: Bootstrap starting.
I20260808 03:57:31.704645 1275 tablet_bootstrap.cc:492] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c: Bootstrap starting.
I20260808 03:57:31.705334 1274 tablet_bootstrap.cc:492] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390: No bootstrap required, opened a new log
I20260808 03:57:31.705502 1274 ts_tablet_manager.cc:1397] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 03:57:31.705636 1273 tablet_bootstrap.cc:654] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:31.706156 1274 raft_consensus.cc:348] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.706375 1274 raft_consensus.cc:374] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:31.706457 1274 raft_consensus.cc:729] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae8ed3767384f0eb0bd7f8d36890390, State: Initialized, Role: FOLLOWER
I20260808 03:57:31.706483 1275 tablet_bootstrap.cc:654] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:31.706718 1274 consensus_queue.cc:260] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.707248 1274 ts_tablet_manager.cc:1428] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260808 03:57:31.709395 1275 tablet_bootstrap.cc:492] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c: No bootstrap required, opened a new log
I20260808 03:57:31.709589 1275 ts_tablet_manager.cc:1397] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260808 03:57:31.710141 1273 tablet_bootstrap.cc:492] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230: No bootstrap required, opened a new log
I20260808 03:57:31.710320 1273 ts_tablet_manager.cc:1397] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260808 03:57:31.710902 1275 raft_consensus.cc:348] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.711187 1275 raft_consensus.cc:374] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:31.711093 1273 raft_consensus.cc:348] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.711275 1275 raft_consensus.cc:729] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a21c0cfa099427890706cc2babfbe5c, State: Initialized, Role: FOLLOWER
I20260808 03:57:31.711350 1273 raft_consensus.cc:374] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:31.711443 1273 raft_consensus.cc:729] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f52ac8f849f4a07b6555ad0ba250230, State: Initialized, Role: FOLLOWER
I20260808 03:57:31.711645 1275 consensus_queue.cc:260] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.711715 1273 consensus_queue.cc:260] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.712333 1273 ts_tablet_manager.cc:1428] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 03:57:31.712872 1275 ts_tablet_manager.cc:1428] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:31.776234 1278 raft_consensus.cc:482] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:31.776516 1278 raft_consensus.cc:504] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.777293 1278 leader_election.cc:288] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981), 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195:44511)
I20260808 03:57:31.777843 955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ca74423eef4206af7cad37ecb8243d" candidate_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f52ac8f849f4a07b6555ad0ba250230" is_pre_election: true
I20260808 03:57:31.777900 468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ca74423eef4206af7cad37ecb8243d" candidate_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a21c0cfa099427890706cc2babfbe5c" is_pre_election: true
I20260808 03:57:31.778115 955 raft_consensus.cc:2454] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ae8ed3767384f0eb0bd7f8d36890390 in term 0.
I20260808 03:57:31.778252 468 raft_consensus.cc:2454] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ae8ed3767384f0eb0bd7f8d36890390 in term 0.
I20260808 03:57:31.778626 592 leader_election.cc:302] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f52ac8f849f4a07b6555ad0ba250230, 3ae8ed3767384f0eb0bd7f8d36890390; no voters:
I20260808 03:57:31.778932 1278 raft_consensus.cc:2781] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:57:31.779044 1278 raft_consensus.cc:482] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:57:31.779134 1278 raft_consensus.cc:3037] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:31.781292 1278 raft_consensus.cc:504] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.781915 1278 leader_election.cc:288] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [CANDIDATE]: Term 1 election: Requested vote from peers 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981), 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195:44511)
I20260808 03:57:31.782400 468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ca74423eef4206af7cad37ecb8243d" candidate_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a21c0cfa099427890706cc2babfbe5c"
I20260808 03:57:31.782442 955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ca74423eef4206af7cad37ecb8243d" candidate_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f52ac8f849f4a07b6555ad0ba250230"
I20260808 03:57:31.782598 468 raft_consensus.cc:3037] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:31.782646 955 raft_consensus.cc:3037] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:31.784881 955 raft_consensus.cc:2454] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ae8ed3767384f0eb0bd7f8d36890390 in term 1.
I20260808 03:57:31.784896 468 raft_consensus.cc:2454] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ae8ed3767384f0eb0bd7f8d36890390 in term 1.
I20260808 03:57:31.785352 591 leader_election.cc:302] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a21c0cfa099427890706cc2babfbe5c, 3ae8ed3767384f0eb0bd7f8d36890390; no voters:
I20260808 03:57:31.785714 1278 raft_consensus.cc:2781] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:31.786119 1278 raft_consensus.cc:686] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 LEADER]: Becoming Leader. State: Replica: 3ae8ed3767384f0eb0bd7f8d36890390, State: Running, Role: LEADER
I20260808 03:57:31.786396 1278 consensus_queue.cc:237] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } }
I20260808 03:57:31.789099 32667 catalog_manager.cc:5495] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194). New cstate: current_term: 1 leader_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ae8ed3767384f0eb0bd7f8d36890390" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 } health_report { overall_health: UNKNOWN } } }
I20260808 03:57:31.809160 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:31.804000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.28.251.194:41777)
0808 03:57:31.804568 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:31.804572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:31.804593 (+ 21us) server_negotiation.cc:407] Connection header received
0808 03:57:31.804653 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:31.804657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:31.804719 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:31.804800 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:31.805130 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:31.805928 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:31.807278 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:31.807829 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:31.807955 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:31.808397 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:31.808553 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:31.808624 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:31.808738 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 03:57:31.808804 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":281,"threads_started":1}
I20260808 03:57:31.818670 955 raft_consensus.cc:1261] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Refusing update from remote peer 3ae8ed3767384f0eb0bd7f8d36890390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:31.818789 468 raft_consensus.cc:1261] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Refusing update from remote peer 3ae8ed3767384f0eb0bd7f8d36890390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:31.819406 1278 consensus_queue.cc:1035] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f52ac8f849f4a07b6555ad0ba250230" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 44511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:31.819768 1303 consensus_queue.cc:1035] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a21c0cfa099427890706cc2babfbe5c" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 03:57:31.842209 1217 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 03:57:33.431674 32665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:33.431939 32665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:33.436349 32665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 865f5d73f42944589e531a509d46d3fc: Sending DeleteTablet for 3 replicas of tablet f5ca74423eef4206af7cad37ecb8243d
I20260808 03:57:33.437255 681 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ca74423eef4206af7cad37ecb8243d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:33 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:37494
I20260808 03:57:33.437255 436 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ca74423eef4206af7cad37ecb8243d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:33 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:45588
I20260808 03:57:33.437768 924 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ca74423eef4206af7cad37ecb8243d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:33 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:33498
I20260808 03:57:33.438764 1346 tablet_replica.cc:330] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390: stopping tablet replica
I20260808 03:57:33.439028 1346 raft_consensus.cc:2229] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:57:33.439154 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 325
I20260808 03:57:33.439581 1346 raft_consensus.cc:2258] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:33.440753 1346 ts_tablet_manager.cc:1905] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:33.444341 1347 tablet_replica.cc:330] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c: stopping tablet replica
I20260808 03:57:33.444561 1348 tablet_replica.cc:330] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230: stopping tablet replica
I20260808 03:57:33.444692 1347 raft_consensus.cc:2229] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:33.444783 1348 raft_consensus.cc:2229] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:33.445122 1348 raft_consensus.cc:2258] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:33.445158 1347 raft_consensus.cc:2258] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:33.446409 1347 ts_tablet_manager.cc:1905] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:33.447116 1348 ts_tablet_manager.cc:1905] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:33.449635 1346 ts_tablet_manager.cc:1918] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 03:57:33.449793 1346 log.cc:1196] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/f5ca74423eef4206af7cad37ecb8243d
I20260808 03:57:33.450269 1346 ts_tablet_manager.cc:1939] T f5ca74423eef4206af7cad37ecb8243d P 3ae8ed3767384f0eb0bd7f8d36890390: Deleting consensus metadata
I20260808 03:57:33.451601 32638 catalog_manager.cc:4841] TS 3ae8ed3767384f0eb0bd7f8d36890390 (127.28.251.194:41777): tablet f5ca74423eef4206af7cad37ecb8243d (table test-table [id=410bc99c58314d5aaa3fa43786685733]) successfully deleted
I20260808 03:57:33.453979 1348 ts_tablet_manager.cc:1918] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 03:57:33.453977 1347 ts_tablet_manager.cc:1918] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 03:57:33.454113 1348 log.cc:1196] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/f5ca74423eef4206af7cad37ecb8243d
I20260808 03:57:33.454123 1347 log.cc:1196] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/f5ca74423eef4206af7cad37ecb8243d
I20260808 03:57:33.454510 1348 ts_tablet_manager.cc:1939] T f5ca74423eef4206af7cad37ecb8243d P 1f52ac8f849f4a07b6555ad0ba250230: Deleting consensus metadata
I20260808 03:57:33.454523 1347 ts_tablet_manager.cc:1939] T f5ca74423eef4206af7cad37ecb8243d P 0a21c0cfa099427890706cc2babfbe5c: Deleting consensus metadata
I20260808 03:57:33.455569 32638 catalog_manager.cc:4841] TS 1f52ac8f849f4a07b6555ad0ba250230 (127.28.251.195:44511): tablet f5ca74423eef4206af7cad37ecb8243d (table test-table [id=410bc99c58314d5aaa3fa43786685733]) successfully deleted
I20260808 03:57:33.455571 32636 catalog_manager.cc:4841] TS 0a21c0cfa099427890706cc2babfbe5c (127.28.251.193:35981): tablet f5ca74423eef4206af7cad37ecb8243d (table test-table [id=410bc99c58314d5aaa3fa43786685733]) successfully deleted
Aug 08 03:57:33 dist-test-slave-pww0 krb5kdc[32574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:57:33.467141 1350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:33.440392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.28.251.193:35981)
0808 03:57:33.444893 (+ 4501us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:33.444900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:33.446340 (+ 1440us) server_negotiation.cc:407] Connection header received
0808 03:57:33.446528 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:33.446533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:33.446616 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:33.446743 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:33.448592 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:33.449776 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:33.451830 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:33.452556 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:33.452762 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:33.461126 (+ 8364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:33.461145 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:33.461163 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:33.461215 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:33.464390 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:33.465127 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:33.465136 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:33.465138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:33.465306 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:33.465703 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:33.465707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:33.465709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:33.466093 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:33.466337 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:33.466567 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 03:57:33.466764 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4345,"thread_start_us":4187,"threads_started":1}
W20260808 03:57:33.467947 546 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55176
W20260808 03:57:33.468583 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 325: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 03:57:33.468813 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 325
I20260808 03:57:33.503588 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 565
Aug 08 03:57:33 dist-test-slave-pww0 krb5kdc[32574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 03:57:33.519122 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:33.504328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.28.251.194:41777)
0808 03:57:33.504829 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:33.504834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:33.504851 (+ 17us) server_negotiation.cc:407] Connection header received
0808 03:57:33.504909 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:33.504913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:33.504977 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:33.505085 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:33.506387 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:33.507167 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:33.508520 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:33.508933 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:33.509030 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:33.513401 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:33.513421 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:33.513438 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:33.513486 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:33.516398 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:33.517175 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:33.517184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:33.517186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:33.517351 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:33.517730 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:33.517733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:33.517735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:33.518126 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:33.518379 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:33.518641 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 03:57:33.518835 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":250,"threads_started":1}
W20260808 03:57:33.520126 801 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58518
W20260808 03:57:33.520810 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 565: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 03:57:33.520961 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 565
I20260808 03:57:33.555111 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 817
Aug 08 03:57:33 dist-test-slave-pww0 krb5kdc[32574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 03:57:33.570787 1353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:33.555894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.28.251.195:44511)
0808 03:57:33.556416 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:33.556421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:33.556437 (+ 16us) server_negotiation.cc:407] Connection header received
0808 03:57:33.556495 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:33.556499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:33.556569 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:33.556660 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:33.557949 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:33.558728 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:33.560197 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:33.560616 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:33.560731 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:33.565195 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:33.565214 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:33.565233 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:33.565296 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:33.568250 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:33.568919 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:33.568927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:33.568930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:33.569076 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:33.569400 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:33.569403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:33.569405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:33.569781 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:33.570006 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:33.570228 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 03:57:33.570434 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":250,"threads_started":1}
W20260808 03:57:33.571585 1036 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57330
W20260808 03:57:33.572221 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 817: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 03:57:33.572387 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 817
I20260808 03:57:33.609344 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 32599
I20260808 03:57:33.610607 32731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:33.731143 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 32599
2026-08-08T03:57:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9005 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:57:33 dist-test-slave-pww0 krb5kdc[1357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:57:33 dist-test-slave-pww0 krb5kdc[1357](info): set up 2 sockets
Aug 08 03:57:33 dist-test-slave-pww0 krb5kdc[1357](info): commencing operation
krb5kdc: starting...
W20260808 03:57:34.973611 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:57:35 dist-test-slave-pww0 krb5kdc[1357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:57:35.224895 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:42551
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:42551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:35.584056 1377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:35.584389 1377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:35.584528 1377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:35.592149 1377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:35.592285 1377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:35.592355 1377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:35.592440 1377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:35.592494 1377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:57:35.603039 1377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:42551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:42551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:35.604564 1377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:35.605959 1377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:35.618680 1383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:35.618708 1388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:35.619210 1382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:35.620028 1377 server_base.cc:1034] running on GCE node
I20260808 03:57:35.620877 1377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:35.622478 1377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:35.623728 1377 hybrid_clock.cc:648] HybridClock initialized: now 1786161455623663 us; error 50 us; skew 500 ppm
I20260808 03:57:35.624115 1377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:35 dist-test-slave-pww0 krb5kdc[1357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161455, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:35.627568 1377 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:57:35.629266 1377 webserver.cc:460] Webserver started at http://127.28.251.254:34069/ using document root <none> and password file <none>
I20260808 03:57:35.630029 1377 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:35.630160 1377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:35.630501 1377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:35.633774 1377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "029fee6133c84b068a2e8921945d9602"
format_stamp: "Formatted at 2026-08-08 03:57:35 on dist-test-slave-pww0"
server_key: "581f99deb46cc26d7d368f1f396b483b"
server_key_iv: "d5c87a3b6acb5a77d1bb0fa6381c968b"
server_key_version: "encryptionkey@0"
I20260808 03:57:35.634581 1377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "029fee6133c84b068a2e8921945d9602"
format_stamp: "Formatted at 2026-08-08 03:57:35 on dist-test-slave-pww0"
server_key: "581f99deb46cc26d7d368f1f396b483b"
server_key_iv: "d5c87a3b6acb5a77d1bb0fa6381c968b"
server_key_version: "encryptionkey@0"
I20260808 03:57:35.640519 1377 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 03:57:35.644690 1402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:35.646350 1377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 03:57:35.646572 1377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "029fee6133c84b068a2e8921945d9602"
format_stamp: "Formatted at 2026-08-08 03:57:35 on dist-test-slave-pww0"
server_key: "581f99deb46cc26d7d368f1f396b483b"
server_key_iv: "d5c87a3b6acb5a77d1bb0fa6381c968b"
server_key_version: "encryptionkey@0"
I20260808 03:57:35.646812 1377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:35.666229 1377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:35.670691 1377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:35.670965 1377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:35.689460 1377 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:42551
I20260808 03:57:35.689492 1489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:42551 every 8 connection(s)
I20260808 03:57:35.691085 1377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:35.695379 1491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:35.700412 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 1377
I20260808 03:57:35.700656 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:35.701305 29679 external_mini_cluster.cc:1428] Setting key 7235b3f49e46e847571ca53513416211
I20260808 03:57:35.709288 1491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602: Bootstrap starting.
I20260808 03:57:35.713413 1491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:35.714620 1491 log.cc:824] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:35.717978 1491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602: No bootstrap required, opened a new log
Aug 08 03:57:35 dist-test-slave-pww0 krb5kdc[1357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:35.727447 1495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:35.703723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.28.251.254:42551)
0808 03:57:35.704663 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:35.704679 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:35.704743 (+ 64us) server_negotiation.cc:407] Connection header received
0808 03:57:35.707384 (+ 2641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:35.707421 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:35.708054 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:35.708567 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:35.710103 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:35.712234 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:35.713670 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:35.714443 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:35.714663 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:35.719157 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:35.719170 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:35.719196 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:35.719245 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:35.722612 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:35.723601 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:35.723609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:35.723617 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:35.723854 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:35.724297 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:35.724300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:35.724302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:35.724944 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:35.725236 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:35.725775 (+ 539us) server_negotiation.cc:299] Negotiation successful
0808 03:57:35.726294 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":411,"threads_started":1}
I20260808 03:57:35.727429 1491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029fee6133c84b068a2e8921945d9602" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42551 } }
I20260808 03:57:35.727974 1491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:35.728086 1491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 029fee6133c84b068a2e8921945d9602, State: Initialized, Role: FOLLOWER
I20260808 03:57:35.728821 1491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029fee6133c84b068a2e8921945d9602" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42551 } }
I20260808 03:57:35.729135 1491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:35.729290 1491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:35.729481 1491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:35.732151 1491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029fee6133c84b068a2e8921945d9602" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42551 } }
I20260808 03:57:35.732674 1491 leader_election.cc:302] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 029fee6133c84b068a2e8921945d9602; no voters:
I20260808 03:57:35.733259 1491 leader_election.cc:288] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:35.733449 1497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:35.734375 1497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [term 1 LEADER]: Becoming Leader. State: Replica: 029fee6133c84b068a2e8921945d9602, State: Running, Role: LEADER
I20260808 03:57:35.734990 1497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029fee6133c84b068a2e8921945d9602" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42551 } }
I20260808 03:57:35.735484 1491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:35.738102 1499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 029fee6133c84b068a2e8921945d9602. Latest consensus state: current_term: 1 leader_uuid: "029fee6133c84b068a2e8921945d9602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029fee6133c84b068a2e8921945d9602" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42551 } } }
I20260808 03:57:35.738525 1499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:35.739096 1498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "029fee6133c84b068a2e8921945d9602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029fee6133c84b068a2e8921945d9602" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 42551 } } }
I20260808 03:57:35.739414 1498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:35.739544 1505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:35.746546 1505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:35.759210 1505 catalog_manager.cc:1347] Generated new cluster ID: 59c66e68168340a98a4fb494a39bcf3b
I20260808 03:57:35.759357 1505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:35.780443 1505 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:35.782027 1505 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:35.791949 1505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 029fee6133c84b068a2e8921945d9602: Generated new TSK 0
I20260808 03:57:35.793239 1505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:57:35.871968 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:42551
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:36.254611 1528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:36.254937 1528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:36.255051 1528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:36.262951 1528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:36.263095 1528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:36.263233 1528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:57:36.274058 1528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:42551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:36.275485 1528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:36.276929 1528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:36.290757 1538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:36.290812 1540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:36.290717 1537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:36.292650 1528 server_base.cc:1034] running on GCE node
I20260808 03:57:36.293637 1528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:36.295089 1528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:36.296356 1528 hybrid_clock.cc:648] HybridClock initialized: now 1786161456296298 us; error 70 us; skew 500 ppm
I20260808 03:57:36.296728 1528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:36 dist-test-slave-pww0 krb5kdc[1357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161456, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:36.300374 1528 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:57:36.302079 1528 webserver.cc:460] Webserver started at http://127.28.251.193:46069/ using document root <none> and password file <none>
I20260808 03:57:36.302875 1528 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:36.303017 1528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:36.303345 1528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:36.306568 1528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "8706705ec53f464bb2084f302f79c8e1"
format_stamp: "Formatted at 2026-08-08 03:57:36 on dist-test-slave-pww0"
server_key: "00583caacec54d941293549fce9e3ae4"
server_key_iv: "13f0a0471993845cec2f0adc8097dc8c"
server_key_version: "encryptionkey@0"
I20260808 03:57:36.307435 1528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "8706705ec53f464bb2084f302f79c8e1"
format_stamp: "Formatted at 2026-08-08 03:57:36 on dist-test-slave-pww0"
server_key: "00583caacec54d941293549fce9e3ae4"
server_key_iv: "13f0a0471993845cec2f0adc8097dc8c"
server_key_version: "encryptionkey@0"
I20260808 03:57:36.313318 1528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 03:57:36.317224 1559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:36.318683 1528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 03:57:36.318883 1528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "8706705ec53f464bb2084f302f79c8e1"
format_stamp: "Formatted at 2026-08-08 03:57:36 on dist-test-slave-pww0"
server_key: "00583caacec54d941293549fce9e3ae4"
server_key_iv: "13f0a0471993845cec2f0adc8097dc8c"
server_key_version: "encryptionkey@0"
I20260808 03:57:36.319109 1528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:36.333218 1528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:36.337987 1528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:36.338299 1528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:36.339428 1528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:36.340998 1528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:36.341085 1528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:36.341218 1528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:36.341275 1528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:36.373781 1528 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:33189
I20260808 03:57:36.373823 1737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:33189 every 8 connection(s)
I20260808 03:57:36.375185 1528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:57:36.382892 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 1528
I20260808 03:57:36.383103 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:57:36.383798 29679 external_mini_cluster.cc:1428] Setting key 2a721680e4ef67be38b97eb5e4b410ce
Aug 08 03:57:36 dist-test-slave-pww0 krb5kdc[1357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161456, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:36.396255 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:36.377854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:42395 (local address 127.28.251.254:42551)
0808 03:57:36.378321 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:36.378326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:36.379341 (+ 1015us) server_negotiation.cc:407] Connection header received
0808 03:57:36.381294 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:36.381299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:36.381395 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:36.381499 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:36.383169 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:36.384024 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:36.385701 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:36.386128 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:36.386246 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:36.390210 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:36.390216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:36.390219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:36.390259 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:36.392864 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:36.393709 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:36.393714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:36.393716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:36.393867 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:36.394332 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:36.394337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:36.394340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:36.394647 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:36.394791 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:36.395631 (+ 840us) server_negotiation.cc:299] Negotiation successful
0808 03:57:36.395897 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260808 03:57:36.397313 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:36.378333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:42551 (local address 127.28.251.193:42395)
0808 03:57:36.379081 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:36.379128 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:36.381004 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:36.381718 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:36.381727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:36.382317 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:36.382944 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:36.382964 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:36.384215 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:36.384219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:36.385539 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:36.385553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:36.386398 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:36.386407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:36.386658 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:36.387378 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:36.387409 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:36.389926 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:36.393043 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:36.393054 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:36.393071 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:36.393570 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:36.393991 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:36.393996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:36.394000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:36.394214 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:36.394790 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:36.394800 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:36.395204 (+ 404us) client_negotiation.cc:769] Sending connection context
0808 03:57:36.395510 (+ 306us) client_negotiation.cc:240] Negotiation successful
0808 03:57:36.396104 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":278,"threads_started":1}
I20260808 03:57:36.399050 1742 heartbeater.cc:347] Connected to a master server at 127.28.251.254:42551
I20260808 03:57:36.399365 1742 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:36.400177 1742 heartbeater.cc:511] Master 127.28.251.254:42551 requested a full tablet report, sending...
I20260808 03:57:36.402691 1430 ts_manager.cc:194] Registered new tserver with Master: 8706705ec53f464bb2084f302f79c8e1 (127.28.251.193:33189)
I20260808 03:57:36.404947 1430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:42395
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:57:36.439155 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:42551
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:36.793697 1748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:36.794021 1748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:36.794152 1748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:36.801757 1748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:36.801900 1748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:36.802048 1748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:57:36.812614 1748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:42551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:36.814060 1748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:36.815596 1748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:36.829967 1760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:36.830029 1755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:36.829928 1753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:36.833139 1748 server_base.cc:1034] running on GCE node
I20260808 03:57:36.833868 1748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:36.835035 1748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:36.836289 1748 hybrid_clock.cc:648] HybridClock initialized: now 1786161456836214 us; error 63 us; skew 500 ppm
I20260808 03:57:36.836627 1748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:36 dist-test-slave-pww0 krb5kdc[1357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161456, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:36.840060 1748 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:57:36.841799 1748 webserver.cc:460] Webserver started at http://127.28.251.194:38925/ using document root <none> and password file <none>
I20260808 03:57:36.842577 1748 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:36.842710 1748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:36.843045 1748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:36.846438 1748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "3a179c2a95144456ab67b392f911f038"
format_stamp: "Formatted at 2026-08-08 03:57:36 on dist-test-slave-pww0"
server_key: "9dcb2c7c259b7b97f9238734afdfd9ca"
server_key_iv: "6607f81342c29c6e4bd18b5259f4b5df"
server_key_version: "encryptionkey@0"
I20260808 03:57:36.847378 1748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "3a179c2a95144456ab67b392f911f038"
format_stamp: "Formatted at 2026-08-08 03:57:36 on dist-test-slave-pww0"
server_key: "9dcb2c7c259b7b97f9238734afdfd9ca"
server_key_iv: "6607f81342c29c6e4bd18b5259f4b5df"
server_key_version: "encryptionkey@0"
I20260808 03:57:36.854031 1748 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260808 03:57:36.858104 1768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:36.859686 1748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:36.859885 1748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "3a179c2a95144456ab67b392f911f038"
format_stamp: "Formatted at 2026-08-08 03:57:36 on dist-test-slave-pww0"
server_key: "9dcb2c7c259b7b97f9238734afdfd9ca"
server_key_iv: "6607f81342c29c6e4bd18b5259f4b5df"
server_key_version: "encryptionkey@0"
I20260808 03:57:36.860122 1748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:36.874749 1748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:36.879549 1748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:36.879877 1748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:36.880947 1748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:36.882448 1748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:36.882532 1748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:36.882663 1748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:36.882723 1748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:36.915752 1748 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:45419
I20260808 03:57:36.915803 1944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:45419 every 8 connection(s)
I20260808 03:57:36.917131 1748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:57:36.925067 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 1748
I20260808 03:57:36.925266 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:57:36.925894 29679 external_mini_cluster.cc:1428] Setting key b7e106560fb151bdd309ad1e85f5f3e0
Aug 08 03:57:36 dist-test-slave-pww0 krb5kdc[1357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161456, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:36.940790 1951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:36.919752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:38895 (local address 127.28.251.254:42551)
0808 03:57:36.920218 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:36.920223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:36.921178 (+ 955us) server_negotiation.cc:407] Connection header received
0808 03:57:36.923188 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:36.923194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:36.923292 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:36.923427 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:36.925277 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:36.926374 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:36.928781 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:36.929420 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:36.929553 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:36.933643 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:36.933651 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:36.933656 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:36.933704 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:36.936994 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:36.937989 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:36.937995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:36.937997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:36.938141 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:36.938721 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:36.938724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:36.938726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:36.938999 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:36.939115 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:36.940140 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0808 03:57:36.940396 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":246,"threads_started":1}
I20260808 03:57:36.941776 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:36.920246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:42551 (local address 127.28.251.194:38895)
0808 03:57:36.920995 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:36.921037 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:36.922885 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:36.923709 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:36.923718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:36.924398 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:36.925057 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:36.925077 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:36.926559 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:36.926565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:36.928610 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:36.928627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:36.929695 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:36.929701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:36.929940 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:36.930742 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:36.930769 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:36.933312 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:36.937219 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:36.937229 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:36.937246 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:36.937811 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:36.938264 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:36.938270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:36.938273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:36.938600 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:36.939141 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:36.939151 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:36.939703 (+ 552us) client_negotiation.cc:769] Sending connection context
0808 03:57:36.939985 (+ 282us) client_negotiation.cc:240] Negotiation successful
0808 03:57:36.940468 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":283,"threads_started":1}
I20260808 03:57:36.943497 1948 heartbeater.cc:347] Connected to a master server at 127.28.251.254:42551
I20260808 03:57:36.943835 1948 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:36.944633 1948 heartbeater.cc:511] Master 127.28.251.254:42551 requested a full tablet report, sending...
I20260808 03:57:36.946223 1430 ts_manager.cc:194] Registered new tserver with Master: 3a179c2a95144456ab67b392f911f038 (127.28.251.194:45419)
I20260808 03:57:36.947111 1430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:38895
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:57:36.982725 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:42551
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:37.345247 1959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:37.345598 1959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:37.345747 1959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:37.354157 1959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:37.354317 1959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:37.354499 1959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:57:37.365438 1959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:42551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:37.367017 1959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:37.368510 1959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:37.383514 1964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:37.383488 1970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:37.383612 1968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:37.384101 1959 server_base.cc:1034] running on GCE node
I20260808 03:57:37.384858 1959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:37.385910 1959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:37.387138 1959 hybrid_clock.cc:648] HybridClock initialized: now 1786161457387069 us; error 55 us; skew 500 ppm
I20260808 03:57:37.387557 1959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:37 dist-test-slave-pww0 krb5kdc[1357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161457, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:37.390908 1959 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:57:37.392525 1959 webserver.cc:460] Webserver started at http://127.28.251.195:40711/ using document root <none> and password file <none>
I20260808 03:57:37.393303 1959 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:37.393429 1959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:37.393765 1959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:37.397024 1959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "f7d77dcc7b6d44f3b117750471515a4b"
format_stamp: "Formatted at 2026-08-08 03:57:37 on dist-test-slave-pww0"
server_key: "6fc5b1da0a3b9310867294c7c8c3be92"
server_key_iv: "1d811312fc7da417063e55c48b2310d6"
server_key_version: "encryptionkey@0"
I20260808 03:57:37.397920 1959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "f7d77dcc7b6d44f3b117750471515a4b"
format_stamp: "Formatted at 2026-08-08 03:57:37 on dist-test-slave-pww0"
server_key: "6fc5b1da0a3b9310867294c7c8c3be92"
server_key_iv: "1d811312fc7da417063e55c48b2310d6"
server_key_version: "encryptionkey@0"
I20260808 03:57:37.403890 1959 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260808 03:57:37.408033 1980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:37.408521 1742 heartbeater.cc:503] Master 127.28.251.254:42551 was elected leader, sending a full tablet report...
I20260808 03:57:37.409740 1959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 03:57:37.409951 1959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "f7d77dcc7b6d44f3b117750471515a4b"
format_stamp: "Formatted at 2026-08-08 03:57:37 on dist-test-slave-pww0"
server_key: "6fc5b1da0a3b9310867294c7c8c3be92"
server_key_iv: "1d811312fc7da417063e55c48b2310d6"
server_key_version: "encryptionkey@0"
I20260808 03:57:37.410197 1959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:37.435877 1959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:37.440917 1959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:37.441226 1959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:37.442305 1959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:37.443810 1959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:37.443892 1959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:37.444013 1959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:37.444067 1959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:37.475906 1959 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:33003
I20260808 03:57:37.475965 2171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:33003 every 8 connection(s)
I20260808 03:57:37.477240 1959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:57:37.479424 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 1959
I20260808 03:57:37.479707 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:57:37.480254 29679 external_mini_cluster.cc:1428] Setting key 45ef9bf02011b93aac58beede2e994b8
Aug 08 03:57:37 dist-test-slave-pww0 krb5kdc[1357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161457, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:37.499802 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:37.479957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:36581 (local address 127.28.251.254:42551)
0808 03:57:37.480427 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:37.480432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:37.481724 (+ 1292us) server_negotiation.cc:407] Connection header received
0808 03:57:37.484169 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:37.484174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:37.484287 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:37.484386 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:37.486040 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:37.486849 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:37.488685 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:37.489117 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:37.489236 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:37.493011 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:37.493018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:37.493021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:37.493063 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:37.496443 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:37.497331 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:37.497335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:37.497338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:37.497471 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:37.497926 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:37.497929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:37.497931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:37.498211 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:37.498355 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:37.499133 (+ 778us) server_negotiation.cc:299] Negotiation successful
0808 03:57:37.499368 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260808 03:57:37.500828 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:37.480663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:42551 (local address 127.28.251.195:36581)
0808 03:57:37.481558 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:37.481624 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:37.483820 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:37.484653 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:37.484662 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:37.485246 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:37.485862 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:37.485881 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:37.487046 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:37.487050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:37.488475 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:37.488493 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:37.489330 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:37.489336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:37.489558 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:37.490247 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:37.490271 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:37.492734 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:37.496619 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:37.496629 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:37.496646 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:37.497195 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:37.497584 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:37.497590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:37.497593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:37.497805 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:37.498350 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:37.498359 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:37.498728 (+ 369us) client_negotiation.cc:769] Sending connection context
0808 03:57:37.498998 (+ 270us) client_negotiation.cc:240] Negotiation successful
0808 03:57:37.499615 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":314,"threads_started":1}
I20260808 03:57:37.502180 2172 heartbeater.cc:347] Connected to a master server at 127.28.251.254:42551
I20260808 03:57:37.502517 2172 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:37.503293 2172 heartbeater.cc:511] Master 127.28.251.254:42551 requested a full tablet report, sending...
I20260808 03:57:37.504873 1430 ts_manager.cc:194] Registered new tserver with Master: f7d77dcc7b6d44f3b117750471515a4b (127.28.251.195:33003)
I20260808 03:57:37.505707 1430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:36581
I20260808 03:57:37.509331 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 03:57:37.553862 2174 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:57:37.550943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.28.251.254:42551)
0808 03:57:37.551162 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:37.551171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:37.551663 (+ 492us) server_negotiation.cc:407] Connection header received
0808 03:57:37.551850 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:37.551854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:37.551931 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:37.552024 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:37.553581 (+ 1557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57998: BlockingRecv error: recv got EOF from 127.0.0.1:57998 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 03:57:37.556988 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1528
W20260808 03:57:37.560194 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1528: Not authorized: Client connection negotiation failed: client connection to 127.28.251.193:33189: server requires authentication, but client does not have Kerberos credentials available
I20260808 03:57:37.560423 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 1528
I20260808 03:57:37.584174 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1748
W20260808 03:57:37.587136 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1748: Not authorized: Client connection negotiation failed: client connection to 127.28.251.194:45419: server requires authentication, but client does not have Kerberos credentials available
I20260808 03:57:37.587337 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 1748
I20260808 03:57:37.612248 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1959
W20260808 03:57:37.615713 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1959: Not authorized: Client connection negotiation failed: client connection to 127.28.251.195:33003: server requires authentication, but client does not have Kerberos credentials available
I20260808 03:57:37.615856 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 1959
I20260808 03:57:37.638666 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1377
I20260808 03:57:37.640036 1482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:37.747597 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 1377
2026-08-08T03:57:37Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4016 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:57:37 dist-test-slave-pww0 krb5kdc[2204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:57:37 dist-test-slave-pww0 krb5kdc[2204](info): set up 2 sockets
Aug 08 03:57:37 dist-test-slave-pww0 krb5kdc[2204](info): commencing operation
krb5kdc: starting...
W20260808 03:57:39.002592 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:57:39 dist-test-slave-pww0 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:57:39.255559 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43191
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:43191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:39.605867 2224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:39.606197 2224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:39.606299 2224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:39.613850 2224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:39.613988 2224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:39.614037 2224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:39.614094 2224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:39.614143 2224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:57:39.624537 2224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:43191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:39.625909 2224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:39.627316 2224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:39.640082 2240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:39.640165 2241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:39.640128 2243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:39.641156 2224 server_base.cc:1034] running on GCE node
I20260808 03:57:39.642052 2224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:39.643714 2224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:39.644953 2224 hybrid_clock.cc:648] HybridClock initialized: now 1786161459644906 us; error 46 us; skew 500 ppm
I20260808 03:57:39.645344 2224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:39 dist-test-slave-pww0 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161459, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:39.652648 2224 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:57:39.654353 2224 webserver.cc:460] Webserver started at http://127.28.251.254:44969/ using document root <none> and password file <none>
I20260808 03:57:39.655117 2224 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:39.655233 2224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:39.655597 2224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:39.658890 2224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "06c011bfaa82469c888618d5bb0e479a"
format_stamp: "Formatted at 2026-08-08 03:57:39 on dist-test-slave-pww0"
server_key: "f38c6e124dd54ccaa6c1d8a6499a51dd"
server_key_iv: "c4254d8937d17f0ccf6305e30bd9185e"
server_key_version: "encryptionkey@0"
I20260808 03:57:39.659849 2224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "06c011bfaa82469c888618d5bb0e479a"
format_stamp: "Formatted at 2026-08-08 03:57:39 on dist-test-slave-pww0"
server_key: "f38c6e124dd54ccaa6c1d8a6499a51dd"
server_key_iv: "c4254d8937d17f0ccf6305e30bd9185e"
server_key_version: "encryptionkey@0"
I20260808 03:57:39.665822 2224 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 03:57:39.669782 2255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:39.671310 2224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 03:57:39.671540 2224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "06c011bfaa82469c888618d5bb0e479a"
format_stamp: "Formatted at 2026-08-08 03:57:39 on dist-test-slave-pww0"
server_key: "f38c6e124dd54ccaa6c1d8a6499a51dd"
server_key_iv: "c4254d8937d17f0ccf6305e30bd9185e"
server_key_version: "encryptionkey@0"
I20260808 03:57:39.671746 2224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:39.684264 2224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:39.688678 2224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:39.688933 2224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:39.707768 2224 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:43191
I20260808 03:57:39.707808 2346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:43191 every 8 connection(s)
I20260808 03:57:39.709314 2224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:39.710619 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 2224
I20260808 03:57:39.710862 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:39.711593 29679 external_mini_cluster.cc:1428] Setting key d9a6443867ff66e08cebf28c63b07bf7
I20260808 03:57:39.714212 2347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 03:57:39 dist-test-slave-pww0 krb5kdc[2204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:39.728470 2347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a: Bootstrap starting.
I20260808 03:57:39.732131 2347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:39.733314 2347 log.cc:824] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:39.736626 2347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a: No bootstrap required, opened a new log
I20260808 03:57:39.736896 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:39.713640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.28.251.254:43191)
0808 03:57:39.714562 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:39.714580 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:39.714636 (+ 56us) server_negotiation.cc:407] Connection header received
0808 03:57:39.717034 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:39.717066 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:39.717638 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:39.718162 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:39.719647 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:39.721765 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:39.723222 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:39.724336 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:39.724555 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:39.728876 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:39.728886 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:39.728902 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:39.728961 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:39.732198 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:39.733074 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:39.733082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:39.733090 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:39.733284 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:39.733747 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:39.733751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:39.733754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:39.734403 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:39.734716 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:39.735256 (+ 540us) server_negotiation.cc:299] Negotiation successful
0808 03:57:39.735789 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":358,"threads_started":1}
I20260808 03:57:39.741394 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 03:57:39.745325 2347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c011bfaa82469c888618d5bb0e479a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43191 } }
I20260808 03:57:39.745764 2347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:39.745870 2347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c011bfaa82469c888618d5bb0e479a, State: Initialized, Role: FOLLOWER
I20260808 03:57:39.746500 2347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c011bfaa82469c888618d5bb0e479a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43191 } }
I20260808 03:57:39.746743 2347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:39.746883 2347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:39.747056 2347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:39.749431 2347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c011bfaa82469c888618d5bb0e479a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43191 } }
I20260808 03:57:39.749993 2347 leader_election.cc:302] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06c011bfaa82469c888618d5bb0e479a; no voters:
I20260808 03:57:39.750653 2347 leader_election.cc:288] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:39.750814 2352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:39.751735 2352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [term 1 LEADER]: Becoming Leader. State: Replica: 06c011bfaa82469c888618d5bb0e479a, State: Running, Role: LEADER
I20260808 03:57:39.752332 2352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c011bfaa82469c888618d5bb0e479a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43191 } }
I20260808 03:57:39.753211 2347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:39.755307 2355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06c011bfaa82469c888618d5bb0e479a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c011bfaa82469c888618d5bb0e479a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43191 } } }
I20260808 03:57:39.755316 2356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06c011bfaa82469c888618d5bb0e479a. Latest consensus state: current_term: 1 leader_uuid: "06c011bfaa82469c888618d5bb0e479a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c011bfaa82469c888618d5bb0e479a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43191 } } }
I20260808 03:57:39.755800 2355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:39.755800 2356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:39.759143 2361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:39.767706 2361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:39.793136 2361 catalog_manager.cc:1347] Generated new cluster ID: 448dd30120a84e7a951e9152957d9a6c
I20260808 03:57:39.793269 2361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:39.815373 2361 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:39.817760 2361 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:39.832267 2361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06c011bfaa82469c888618d5bb0e479a: Generated new TSK 0
I20260808 03:57:39.833782 2361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 03:57:39.844246 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:39.824559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.28.251.254:43191)
0808 03:57:39.824743 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:39.824750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:39.835370 (+ 10620us) server_negotiation.cc:407] Connection header received
0808 03:57:39.836011 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:39.836017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:39.836123 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:39.836223 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:39.837892 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:39.839074 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:39.840515 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:39.841214 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:39.841378 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:39.842440 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:39.842447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:39.842451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:39.843358 (+ 907us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:39.843445 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:39.843516 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:39.843632 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:39.843741 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 03:57:39.843861 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 03:57:39.852298 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2224
I20260808 03:57:39.853778 2342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:39.947607 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 2224
2026-08-08T03:57:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2193 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:57:40 dist-test-slave-pww0 krb5kdc[2404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:57:40 dist-test-slave-pww0 krb5kdc[2404](info): set up 2 sockets
Aug 08 03:57:40 dist-test-slave-pww0 krb5kdc[2404](info): commencing operation
krb5kdc: starting...
W20260808 03:57:41.196187 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.008s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:57:41 dist-test-slave-pww0 krb5kdc[2404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:57:41.466759 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:37873
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:41.827821 2428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:41.828177 2428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:41.828310 2428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:41.836162 2428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:41.836310 2428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:41.836395 2428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:41.836480 2428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:41.836540 2428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:57:41.847396 2428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:37873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:37873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:41.848969 2428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:41.850453 2428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:41.863164 2436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:41.863226 2437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:41.863230 2441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:41.864146 2428 server_base.cc:1034] running on GCE node
I20260808 03:57:41.864981 2428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:41.866536 2428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:41.867765 2428 hybrid_clock.cc:648] HybridClock initialized: now 1786161461867699 us; error 46 us; skew 500 ppm
I20260808 03:57:41.868146 2428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:41 dist-test-slave-pww0 krb5kdc[2404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161461, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:41.871826 2428 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:57:41.873615 2428 webserver.cc:460] Webserver started at http://127.28.251.254:42199/ using document root <none> and password file <none>
I20260808 03:57:41.874373 2428 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:41.874509 2428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:41.874846 2428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:41.878203 2428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "6ada70e6eb2b43a3b3548af9751e27d2"
format_stamp: "Formatted at 2026-08-08 03:57:41 on dist-test-slave-pww0"
server_key: "27d4dc96f7b2156ee4508a0cbc836a87"
server_key_iv: "7794965eceab9491b2a88fce4259ec41"
server_key_version: "encryptionkey@0"
I20260808 03:57:41.879031 2428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "6ada70e6eb2b43a3b3548af9751e27d2"
format_stamp: "Formatted at 2026-08-08 03:57:41 on dist-test-slave-pww0"
server_key: "27d4dc96f7b2156ee4508a0cbc836a87"
server_key_iv: "7794965eceab9491b2a88fce4259ec41"
server_key_version: "encryptionkey@0"
I20260808 03:57:41.884755 2428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 03:57:41.888684 2453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:41.890327 2428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 03:57:41.890542 2428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "6ada70e6eb2b43a3b3548af9751e27d2"
format_stamp: "Formatted at 2026-08-08 03:57:41 on dist-test-slave-pww0"
server_key: "27d4dc96f7b2156ee4508a0cbc836a87"
server_key_iv: "7794965eceab9491b2a88fce4259ec41"
server_key_version: "encryptionkey@0"
I20260808 03:57:41.890769 2428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:41.904428 2428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:41.908941 2428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:41.909229 2428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:41.927671 2428 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:37873
I20260808 03:57:41.927690 2533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:37873 every 8 connection(s)
I20260808 03:57:41.929176 2428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:41.932994 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 2428
I20260808 03:57:41.933221 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:41.933323 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:41.934072 29679 external_mini_cluster.cc:1428] Setting key 0dfef6bcdd983f44ce7aa02696a940ad
I20260808 03:57:41.948886 2535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2: Bootstrap starting.
Aug 08 03:57:41 dist-test-slave-pww0 krb5kdc[2404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:41.952648 2535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:41.953785 2535 log.cc:824] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:41.957203 2535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2: No bootstrap required, opened a new log
I20260808 03:57:41.958973 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:41.936221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.28.251.254:37873)
0808 03:57:41.937079 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:41.937091 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:41.937136 (+ 45us) server_negotiation.cc:407] Connection header received
0808 03:57:41.938717 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:41.938741 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:41.939168 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:41.939602 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:41.941438 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:41.942767 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:41.944589 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:41.945417 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:41.945593 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:41.950556 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:41.950567 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:41.950582 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:41.950632 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:41.953989 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:41.955006 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:41.955016 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:41.955027 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:41.955240 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:41.955883 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:41.955888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:41.955890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:41.956521 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:41.956877 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:41.957401 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 03:57:41.957895 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":379,"threads_started":1}
I20260808 03:57:41.965721 2535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada70e6eb2b43a3b3548af9751e27d2" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37873 } }
I20260808 03:57:41.966194 2535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:41.966307 2535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ada70e6eb2b43a3b3548af9751e27d2, State: Initialized, Role: FOLLOWER
I20260808 03:57:41.967007 2535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada70e6eb2b43a3b3548af9751e27d2" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37873 } }
I20260808 03:57:41.967262 2535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:41.967401 2535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:41.967622 2535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:41.970237 2535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada70e6eb2b43a3b3548af9751e27d2" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37873 } }
I20260808 03:57:41.970819 2535 leader_election.cc:302] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ada70e6eb2b43a3b3548af9751e27d2; no voters:
I20260808 03:57:41.971649 2535 leader_election.cc:288] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:41.971853 2541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:41.972779 2541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [term 1 LEADER]: Becoming Leader. State: Replica: 6ada70e6eb2b43a3b3548af9751e27d2, State: Running, Role: LEADER
I20260808 03:57:41.973495 2541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada70e6eb2b43a3b3548af9751e27d2" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37873 } }
I20260808 03:57:41.974068 2535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:41.976781 2542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ada70e6eb2b43a3b3548af9751e27d2. Latest consensus state: current_term: 1 leader_uuid: "6ada70e6eb2b43a3b3548af9751e27d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada70e6eb2b43a3b3548af9751e27d2" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37873 } } }
I20260808 03:57:41.977260 2542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:41.977768 2544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ada70e6eb2b43a3b3548af9751e27d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada70e6eb2b43a3b3548af9751e27d2" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37873 } } }
I20260808 03:57:41.978097 2544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:41.980059 2550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:41.985891 2550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:41.997126 2550 catalog_manager.cc:1347] Generated new cluster ID: ccca50438cf64ad9b999179384546ef5
I20260808 03:57:41.997264 2550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:42.017807 2550 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:42.019295 2550 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:42.029557 2550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ada70e6eb2b43a3b3548af9751e27d2: Generated new TSK 0
I20260808 03:57:42.030857 2550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:57:42.095191 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37873
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:42.456612 2569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:42.456926 2569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:42.457049 2569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:42.464821 2569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:42.464965 2569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:42.465113 2569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:57:42.475762 2569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:42.477174 2569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:42.478582 2569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:42.492470 2581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:42.492501 2585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:42.492537 2582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:42.493551 2569 server_base.cc:1034] running on GCE node
I20260808 03:57:42.494259 2569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:42.495334 2569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:42.496562 2569 hybrid_clock.cc:648] HybridClock initialized: now 1786161462496484 us; error 69 us; skew 500 ppm
I20260808 03:57:42.496920 2569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:42 dist-test-slave-pww0 krb5kdc[2404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161462, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:42.500385 2569 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:57:42.501963 2569 webserver.cc:460] Webserver started at http://127.28.251.193:39985/ using document root <none> and password file <none>
I20260808 03:57:42.502704 2569 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:42.502835 2569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:42.503149 2569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:42.506294 2569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "7cd0975bad4f496fbad1d6e1e63d2c96"
format_stamp: "Formatted at 2026-08-08 03:57:42 on dist-test-slave-pww0"
server_key: "43ddf70cf7edf999af7aeab3efd6c784"
server_key_iv: "ac61a850d7f5717911864448f66b1c2d"
server_key_version: "encryptionkey@0"
I20260808 03:57:42.507148 2569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "7cd0975bad4f496fbad1d6e1e63d2c96"
format_stamp: "Formatted at 2026-08-08 03:57:42 on dist-test-slave-pww0"
server_key: "43ddf70cf7edf999af7aeab3efd6c784"
server_key_iv: "ac61a850d7f5717911864448f66b1c2d"
server_key_version: "encryptionkey@0"
I20260808 03:57:42.512773 2569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 03:57:42.516521 2594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:42.517968 2569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 03:57:42.518143 2569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "7cd0975bad4f496fbad1d6e1e63d2c96"
format_stamp: "Formatted at 2026-08-08 03:57:42 on dist-test-slave-pww0"
server_key: "43ddf70cf7edf999af7aeab3efd6c784"
server_key_iv: "ac61a850d7f5717911864448f66b1c2d"
server_key_version: "encryptionkey@0"
I20260808 03:57:42.518362 2569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:42.572718 2569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:42.577344 2569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:42.577669 2569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:42.578743 2569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:42.580367 2569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:42.580456 2569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:42.580581 2569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:42.580636 2569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:42.612993 2569 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:43055
I20260808 03:57:42.613077 2789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:43055 every 8 connection(s)
I20260808 03:57:42.614362 2569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:57:42.622495 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 2569
I20260808 03:57:42.622704 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:57:42.623446 29679 external_mini_cluster.cc:1428] Setting key 69f7dd26ddc7d3b38550c099c5fcedae
Aug 08 03:57:42 dist-test-slave-pww0 krb5kdc[2404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161462, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:42.636283 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:42.617067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:56801 (local address 127.28.251.254:37873)
0808 03:57:42.617523 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:42.617527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:42.618587 (+ 1060us) server_negotiation.cc:407] Connection header received
0808 03:57:42.620464 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:42.620469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:42.620544 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:42.620650 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:42.622310 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:42.623255 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:42.625303 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:42.625898 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:42.626011 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:42.630364 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:42.630370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:42.630373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:42.630413 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:42.633007 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:42.633824 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:42.633829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:42.633831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:42.633956 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:42.634420 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:42.634424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:42.634426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:42.634720 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:42.634856 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:42.635654 (+ 798us) server_negotiation.cc:299] Negotiation successful
0808 03:57:42.635905 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":236,"threads_started":1}
I20260808 03:57:42.637391 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:42.617692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37873 (local address 127.28.251.193:56801)
0808 03:57:42.618417 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:42.618458 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:42.620164 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:42.620851 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:42.620859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:42.621402 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:42.622082 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:42.622116 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:42.623441 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:42.623446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:42.625125 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:42.625139 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:42.626201 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:42.626211 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:42.626550 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:42.627287 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:42.627319 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:42.630091 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:42.633180 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:42.633192 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:42.633209 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:42.633684 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:42.634064 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:42.634069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:42.634073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:42.634299 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:42.634851 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:42.634859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:42.635237 (+ 378us) client_negotiation.cc:769] Sending connection context
0808 03:57:42.635536 (+ 299us) client_negotiation.cc:240] Negotiation successful
0808 03:57:42.636175 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":267,"threads_started":1}
I20260808 03:57:42.639381 2791 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37873
I20260808 03:57:42.639760 2791 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:42.640615 2791 heartbeater.cc:511] Master 127.28.251.254:37873 requested a full tablet report, sending...
I20260808 03:57:42.643251 2484 ts_manager.cc:194] Registered new tserver with Master: 7cd0975bad4f496fbad1d6e1e63d2c96 (127.28.251.193:43055)
I20260808 03:57:42.645576 2484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:56801
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:57:42.683372 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37873
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:43.048727 2803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:43.049060 2803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:43.049190 2803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:43.056828 2803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:43.056962 2803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:43.057109 2803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:57:43.067528 2803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:43.069016 2803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:43.070452 2803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:43.084941 2814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:43.084861 2821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:43.084982 2812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:43.086051 2803 server_base.cc:1034] running on GCE node
I20260808 03:57:43.086750 2803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:43.087999 2803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:43.089253 2803 hybrid_clock.cc:648] HybridClock initialized: now 1786161463089188 us; error 52 us; skew 500 ppm
I20260808 03:57:43.089620 2803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:43 dist-test-slave-pww0 krb5kdc[2404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161463, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:43.093303 2803 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:57:43.095155 2803 webserver.cc:460] Webserver started at http://127.28.251.194:35709/ using document root <none> and password file <none>
I20260808 03:57:43.096005 2803 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:43.096154 2803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:43.096488 2803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:43.099800 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "028bd36ae4cc4c4a883209d2dabb39f0"
format_stamp: "Formatted at 2026-08-08 03:57:43 on dist-test-slave-pww0"
server_key: "4780dbe99bb88bdaa509453bc53fdbf4"
server_key_iv: "525c19ceff0fdf28777370e9f5957c20"
server_key_version: "encryptionkey@0"
I20260808 03:57:43.100677 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "028bd36ae4cc4c4a883209d2dabb39f0"
format_stamp: "Formatted at 2026-08-08 03:57:43 on dist-test-slave-pww0"
server_key: "4780dbe99bb88bdaa509453bc53fdbf4"
server_key_iv: "525c19ceff0fdf28777370e9f5957c20"
server_key_version: "encryptionkey@0"
I20260808 03:57:43.106678 2803 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260808 03:57:43.110841 2840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:43.112706 2803 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 03:57:43.112933 2803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "028bd36ae4cc4c4a883209d2dabb39f0"
format_stamp: "Formatted at 2026-08-08 03:57:43 on dist-test-slave-pww0"
server_key: "4780dbe99bb88bdaa509453bc53fdbf4"
server_key_iv: "525c19ceff0fdf28777370e9f5957c20"
server_key_version: "encryptionkey@0"
I20260808 03:57:43.113157 2803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:43.131579 2803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:43.136346 2803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:43.136674 2803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:43.137713 2803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:43.139324 2803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:43.139412 2803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:43.139570 2803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:43.139631 2803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:43.173565 2803 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:33779
I20260808 03:57:43.173611 3023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:33779 every 8 connection(s)
I20260808 03:57:43.175060 2803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:57:43.180219 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 2803
I20260808 03:57:43.180416 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:57:43.181046 29679 external_mini_cluster.cc:1428] Setting key 6daaf1c3b192a1f08f236f11ef15f1de
Aug 08 03:57:43 dist-test-slave-pww0 krb5kdc[2404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161463, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:43.200103 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:43.178048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:57243 (local address 127.28.251.254:37873)
0808 03:57:43.178533 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:43.178538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:43.179508 (+ 970us) server_negotiation.cc:407] Connection header received
0808 03:57:43.182104 (+ 2596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:43.182109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:43.182204 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:43.182305 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:43.184376 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.185113 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.187343 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.187838 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.187940 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:43.192698 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:43.192705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:43.192710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:43.192756 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:43.195896 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.197004 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.197009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.197012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.197158 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.197775 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.197779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.197781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.198088 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:43.198205 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:43.199423 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0808 03:57:43.199772 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":259,"threads_started":1}
I20260808 03:57:43.201407 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:43.178532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37873 (local address 127.28.251.194:57243)
0808 03:57:43.179309 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:43.179358 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:43.181793 (+ 2435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:43.182603 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:43.182612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:43.183225 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:43.184170 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:43.184198 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.185300 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.185305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:43.187173 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:43.187191 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.188085 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.188093 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:43.188416 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:43.189189 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:43.189220 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:43.192376 (+ 3156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:43.196092 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:43.196103 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:43.196122 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:43.196828 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:43.197281 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:43.197286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:43.197289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:43.197644 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:43.198225 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:43.198233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:43.198778 (+ 545us) client_negotiation.cc:769] Sending connection context
0808 03:57:43.199214 (+ 436us) client_negotiation.cc:240] Negotiation successful
0808 03:57:43.199859 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":292,"threads_started":1}
I20260808 03:57:43.203073 3027 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37873
I20260808 03:57:43.203433 3027 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:43.204524 3027 heartbeater.cc:511] Master 127.28.251.254:37873 requested a full tablet report, sending...
I20260808 03:57:43.206311 2484 ts_manager.cc:194] Registered new tserver with Master: 028bd36ae4cc4c4a883209d2dabb39f0 (127.28.251.194:33779)
I20260808 03:57:43.207157 2484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:57243
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:57:43.239630 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37873
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:43.598322 3034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:43.598675 3034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:43.598811 3034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:43.606462 3034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:43.606614 3034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:43.606776 3034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:57:43.620018 3034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:43.621377 3034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:43.622783 3034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:43.636665 3043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:43.636690 3048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:43.636726 3042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:43.637635 3034 server_base.cc:1034] running on GCE node
I20260808 03:57:43.638378 3034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:43.639497 3034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:43.640728 3034 hybrid_clock.cc:648] HybridClock initialized: now 1786161463640642 us; error 76 us; skew 500 ppm
I20260808 03:57:43.641077 3034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:43 dist-test-slave-pww0 krb5kdc[2404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161463, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:43.644593 3034 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:57:43.646250 3034 webserver.cc:460] Webserver started at http://127.28.251.195:40801/ using document root <none> and password file <none>
I20260808 03:57:43.647001 3034 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:43.647135 3034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:43.647511 3034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:43.649878 2791 heartbeater.cc:503] Master 127.28.251.254:37873 was elected leader, sending a full tablet report...
I20260808 03:57:43.650970 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "58e57ff5172d4b649128ffa249fd690d"
format_stamp: "Formatted at 2026-08-08 03:57:43 on dist-test-slave-pww0"
server_key: "781e5e74656dc5ab2a103f1a0e6b06a1"
server_key_iv: "5be6a902860aeecdd8ce9d0a66cdfdf3"
server_key_version: "encryptionkey@0"
I20260808 03:57:43.651962 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "58e57ff5172d4b649128ffa249fd690d"
format_stamp: "Formatted at 2026-08-08 03:57:43 on dist-test-slave-pww0"
server_key: "781e5e74656dc5ab2a103f1a0e6b06a1"
server_key_iv: "5be6a902860aeecdd8ce9d0a66cdfdf3"
server_key_version: "encryptionkey@0"
I20260808 03:57:43.657898 3034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260808 03:57:43.662019 3057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:43.663784 3034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:43.663980 3034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "58e57ff5172d4b649128ffa249fd690d"
format_stamp: "Formatted at 2026-08-08 03:57:43 on dist-test-slave-pww0"
server_key: "781e5e74656dc5ab2a103f1a0e6b06a1"
server_key_iv: "5be6a902860aeecdd8ce9d0a66cdfdf3"
server_key_version: "encryptionkey@0"
I20260808 03:57:43.664211 3034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:43.681905 3034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:43.686501 3034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:43.686802 3034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:43.687889 3034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:43.689435 3034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:43.689512 3034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:43.689630 3034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:43.689683 3034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:43.722570 3034 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:45419
I20260808 03:57:43.722627 3225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:45419 every 8 connection(s)
I20260808 03:57:43.724084 3034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:57:43.725864 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 3034
I20260808 03:57:43.726065 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:57:43.726639 29679 external_mini_cluster.cc:1428] Setting key 5234745e4f47ef81003a153024412c8b
Aug 08 03:57:43 dist-test-slave-pww0 krb5kdc[2404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161463, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:43.745195 3229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:43.727089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:41031 (local address 127.28.251.254:37873)
0808 03:57:43.727670 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:43.727676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:43.728939 (+ 1263us) server_negotiation.cc:407] Connection header received
0808 03:57:43.731003 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:43.731008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:43.731091 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:43.731183 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:43.732788 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.733574 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.735134 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.735579 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.735674 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:43.739238 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:43.739244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:43.739247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:43.739285 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:43.742238 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.743071 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.743075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.743078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.743210 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.743647 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.743651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.743653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.743946 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:43.744066 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:43.744725 (+ 659us) server_negotiation.cc:299] Negotiation successful
0808 03:57:43.744910 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":263,"threads_started":1}
I20260808 03:57:43.746285 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:43.727745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37873 (local address 127.28.251.195:41031)
0808 03:57:43.728732 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:43.728818 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:43.730740 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:43.731394 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:43.731403 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:43.732020 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:43.732631 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:43.732651 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.733700 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.733703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:43.735018 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:43.735032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.735769 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.735795 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:43.736017 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:43.736651 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:43.736675 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:43.739002 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:43.742394 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:43.742402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:43.742417 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:43.742961 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:43.743298 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:43.743301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:43.743303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:43.743553 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:43.744043 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:43.744050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:43.744406 (+ 356us) client_negotiation.cc:769] Sending connection context
0808 03:57:43.744641 (+ 235us) client_negotiation.cc:240] Negotiation successful
0808 03:57:43.745076 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":370,"threads_started":1}
I20260808 03:57:43.747871 3226 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37873
I20260808 03:57:43.748180 3226 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:43.749003 3226 heartbeater.cc:511] Master 127.28.251.254:37873 requested a full tablet report, sending...
I20260808 03:57:43.750542 2484 ts_manager.cc:194] Registered new tserver with Master: 58e57ff5172d4b649128ffa249fd690d (127.28.251.195:45419)
I20260808 03:57:43.751366 2484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:41031
I20260808 03:57:43.755033 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 03:57:43 dist-test-slave-pww0 krb5kdc[2404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161463, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 03:57:43 dist-test-slave-pww0 krb5kdc[2404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161463, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:43.819125 3229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:43.804107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.28.251.254:37873)
0808 03:57:43.804323 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:43.804330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:43.804817 (+ 487us) server_negotiation.cc:407] Connection header received
0808 03:57:43.805050 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:43.805054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:43.805139 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:43.805229 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:43.806744 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.807547 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.808852 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.809298 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.809427 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:43.813519 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:43.813526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:43.813529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:43.813574 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:43.816243 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.816991 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.816996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.816998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.817141 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.817569 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.817573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.817575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.817957 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:43.818227 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:43.818569 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 03:57:43.818817 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 03:57:43.819782 2484 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56644
I20260808 03:57:43.821272 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2569
Aug 08 03:57:43 dist-test-slave-pww0 krb5kdc[2404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161463, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:57:43.839044 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:43.822660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.28.251.193:43055)
0808 03:57:43.823305 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:43.823312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:43.823346 (+ 34us) server_negotiation.cc:407] Connection header received
0808 03:57:43.823415 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:43.823421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:43.823732 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:43.823870 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:43.825070 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.826123 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.827627 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.828303 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.828421 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:43.833025 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:43.833042 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:43.833058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:43.833104 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:43.836129 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.836939 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.836948 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.836950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.837092 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.837468 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.837473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.837475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.837899 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:43.838163 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:43.838473 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 03:57:43.838732 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":296,"threads_started":1}
W20260808 03:57:43.840602 2784 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43462
W20260808 03:57:43.842058 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 03:57:43.842381 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 2569
I20260808 03:57:43.868105 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2803
Aug 08 03:57:43 dist-test-slave-pww0 krb5kdc[2404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161463, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 03:57:43.885172 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:43.869102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.28.251.194:33779)
0808 03:57:43.869741 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:43.869748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:43.869781 (+ 33us) server_negotiation.cc:407] Connection header received
0808 03:57:43.869844 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:43.869851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:43.870069 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:43.870207 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:43.871629 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.872737 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.874090 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.874677 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.874829 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:43.879217 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:43.879233 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:43.879250 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:43.879296 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:43.882212 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.882947 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.882955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.882958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.883095 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.883406 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.883409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.883411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.883970 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:43.884220 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:43.884544 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 03:57:43.884769 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":318,"threads_started":1}
W20260808 03:57:43.886521 3018 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49084
W20260808 03:57:43.887735 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2803: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 03:57:43.887895 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 2803
I20260808 03:57:43.918391 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 3034
Aug 08 03:57:43 dist-test-slave-pww0 krb5kdc[2404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161463, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 03:57:43.936431 3243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:43.919400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.28.251.195:45419)
0808 03:57:43.920105 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:43.920112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:43.920145 (+ 33us) server_negotiation.cc:407] Connection header received
0808 03:57:43.920211 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:43.920217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:43.920448 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:43.920576 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:43.922084 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.923191 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.924799 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:43.925375 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:43.925493 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:43.930267 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:43.930283 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:43.930300 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:43.930344 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:43.933473 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.934346 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.934353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.934355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.934482 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:43.934861 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:43.934864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:43.934866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:43.935249 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:43.935544 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:43.935912 (+ 368us) server_negotiation.cc:299] Negotiation successful
0808 03:57:43.936138 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":366,"threads_started":1}
W20260808 03:57:43.937758 3220 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35752
W20260808 03:57:43.938912 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 3034: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 03:57:43.939081 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 3034
I20260808 03:57:43.966780 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2428
I20260808 03:57:43.968088 2527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:44.075943 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 2428
2026-08-08T03:57:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4135 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:57:44 dist-test-slave-pww0 krb5kdc[3251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:57:44 dist-test-slave-pww0 krb5kdc[3251](info): set up 2 sockets
Aug 08 03:57:44 dist-test-slave-pww0 krb5kdc[3251](info): commencing operation
krb5kdc: starting...
W20260808 03:57:45.320809 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:57:45 dist-test-slave-pww0 krb5kdc[3251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:57:45.579443 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:36391
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:36391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:45.932085 3271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:45.932438 3271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:45.932575 3271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:45.940129 3271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:45.940272 3271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:45.940353 3271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:45.940438 3271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:45.940498 3271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:57:45.950903 3271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:36391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:36391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:45.952427 3271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:45.953824 3271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:45.966485 3280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:45.966522 3282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:45.966554 3284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:45.967496 3271 server_base.cc:1034] running on GCE node
I20260808 03:57:45.968348 3271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:45.970021 3271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:45.971268 3271 hybrid_clock.cc:648] HybridClock initialized: now 1786161465971218 us; error 37 us; skew 500 ppm
I20260808 03:57:45.971737 3271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:45 dist-test-slave-pww0 krb5kdc[3251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161465, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:45.977871 3271 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:57:45.979737 3271 webserver.cc:460] Webserver started at http://127.28.251.254:37047/ using document root <none> and password file <none>
I20260808 03:57:45.980544 3271 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:45.980677 3271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:45.981012 3271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:45.984324 3271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "440851af6ac840df9dc689a2ed8f0f33"
format_stamp: "Formatted at 2026-08-08 03:57:45 on dist-test-slave-pww0"
server_key: "28496fdf797ceda2e5024244869f8a7a"
server_key_iv: "622795acf7329a27a57ed7325edd8524"
server_key_version: "encryptionkey@0"
I20260808 03:57:45.985160 3271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "440851af6ac840df9dc689a2ed8f0f33"
format_stamp: "Formatted at 2026-08-08 03:57:45 on dist-test-slave-pww0"
server_key: "28496fdf797ceda2e5024244869f8a7a"
server_key_iv: "622795acf7329a27a57ed7325edd8524"
server_key_version: "encryptionkey@0"
I20260808 03:57:45.990732 3271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260808 03:57:45.994678 3293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:45.996182 3271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:45.996361 3271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "440851af6ac840df9dc689a2ed8f0f33"
format_stamp: "Formatted at 2026-08-08 03:57:45 on dist-test-slave-pww0"
server_key: "28496fdf797ceda2e5024244869f8a7a"
server_key_iv: "622795acf7329a27a57ed7325edd8524"
server_key_version: "encryptionkey@0"
I20260808 03:57:45.996555 3271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:46.015110 3271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:46.019896 3271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:46.020200 3271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:46.039418 3271 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:36391
I20260808 03:57:46.039484 3383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:36391 every 8 connection(s)
I20260808 03:57:46.040980 3271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:46.045185 3389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:46.045668 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 3271
I20260808 03:57:46.045905 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:46.046583 29679 external_mini_cluster.cc:1428] Setting key 026345f55356c788cf28686eacb5a050
I20260808 03:57:46.060405 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33: Bootstrap starting.
Aug 08 03:57:46 dist-test-slave-pww0 krb5kdc[3251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:46.064098 3389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:46.065616 3389 log.cc:824] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:46.069458 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33: No bootstrap required, opened a new log
I20260808 03:57:46.072865 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:46.048852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.28.251.254:36391)
0808 03:57:46.049715 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:46.049732 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:46.049807 (+ 75us) server_negotiation.cc:407] Connection header received
0808 03:57:46.052218 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:46.052253 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:46.052877 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:46.053386 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:46.054873 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:46.056938 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:46.058279 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:46.059369 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:46.059600 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:46.064115 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:46.064129 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:46.064152 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:46.064216 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:46.068194 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:46.069098 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:46.069108 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:46.069117 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:46.069353 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:46.069744 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:46.069748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:46.069750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:46.070391 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:46.070730 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:46.071222 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 03:57:46.071722 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":340,"threads_started":1}
I20260808 03:57:46.078339 3389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440851af6ac840df9dc689a2ed8f0f33" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36391 } }
I20260808 03:57:46.078776 3389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:46.078887 3389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440851af6ac840df9dc689a2ed8f0f33, State: Initialized, Role: FOLLOWER
I20260808 03:57:46.079521 3389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440851af6ac840df9dc689a2ed8f0f33" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36391 } }
I20260808 03:57:46.079809 3389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:46.079964 3389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:46.080158 3389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:46.082736 3389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440851af6ac840df9dc689a2ed8f0f33" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36391 } }
I20260808 03:57:46.083302 3389 leader_election.cc:302] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 440851af6ac840df9dc689a2ed8f0f33; no voters:
I20260808 03:57:46.084059 3389 leader_election.cc:288] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:46.084173 3396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:46.084929 3396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [term 1 LEADER]: Becoming Leader. State: Replica: 440851af6ac840df9dc689a2ed8f0f33, State: Running, Role: LEADER
I20260808 03:57:46.085561 3396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440851af6ac840df9dc689a2ed8f0f33" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36391 } }
I20260808 03:57:46.086217 3389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:46.088500 3397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "440851af6ac840df9dc689a2ed8f0f33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440851af6ac840df9dc689a2ed8f0f33" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36391 } } }
I20260808 03:57:46.088922 3397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:46.094417 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 440851af6ac840df9dc689a2ed8f0f33. Latest consensus state: current_term: 1 leader_uuid: "440851af6ac840df9dc689a2ed8f0f33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "440851af6ac840df9dc689a2ed8f0f33" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36391 } } }
I20260808 03:57:46.094795 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33 [sys.catalog]: This master's current role is: LEADER
W20260808 03:57:46.098997 3418 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 03:57:46.099133 3418 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 03:57:46.099257 3407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:46.101076 3407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:46.110061 3407 catalog_manager.cc:1347] Generated new cluster ID: 089e84f64f3242b0b9513d720602c241
I20260808 03:57:46.110209 3407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:46.131493 3407 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:46.133566 3407 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:46.156934 3407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 440851af6ac840df9dc689a2ed8f0f33: Generated new TSK 0
I20260808 03:57:46.158224 3407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:57:46.230731 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36391
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:46.585733 3425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:46.586040 3425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:46.586174 3425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:46.593889 3425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:46.594017 3425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:46.594143 3425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:57:46.604727 3425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:46.606143 3425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:46.607548 3425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:46.621302 3432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:46.621302 3436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:46.621840 3431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:46.622393 3425 server_base.cc:1034] running on GCE node
I20260808 03:57:46.623055 3425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:46.624143 3425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:46.625366 3425 hybrid_clock.cc:648] HybridClock initialized: now 1786161466625304 us; error 53 us; skew 500 ppm
I20260808 03:57:46.625695 3425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:46 dist-test-slave-pww0 krb5kdc[3251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161466, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:46.629117 3425 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:57:46.630676 3425 webserver.cc:460] Webserver started at http://127.28.251.193:40117/ using document root <none> and password file <none>
I20260808 03:57:46.631407 3425 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:46.631572 3425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:46.631898 3425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:46.635003 3425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "425a2b7fa02643fbba48795201649603"
format_stamp: "Formatted at 2026-08-08 03:57:46 on dist-test-slave-pww0"
server_key: "a84af3b9bc95283e5f39581f6eef1783"
server_key_iv: "51c93146af8267257e158cca5c138f7b"
server_key_version: "encryptionkey@0"
I20260808 03:57:46.635887 3425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "425a2b7fa02643fbba48795201649603"
format_stamp: "Formatted at 2026-08-08 03:57:46 on dist-test-slave-pww0"
server_key: "a84af3b9bc95283e5f39581f6eef1783"
server_key_iv: "51c93146af8267257e158cca5c138f7b"
server_key_version: "encryptionkey@0"
I20260808 03:57:46.641492 3425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 03:57:46.645537 3447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:46.647158 3425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 03:57:46.647349 3425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "425a2b7fa02643fbba48795201649603"
format_stamp: "Formatted at 2026-08-08 03:57:46 on dist-test-slave-pww0"
server_key: "a84af3b9bc95283e5f39581f6eef1783"
server_key_iv: "51c93146af8267257e158cca5c138f7b"
server_key_version: "encryptionkey@0"
I20260808 03:57:46.647595 3425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:46.685286 3425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:46.689841 3425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:46.690146 3425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:46.691229 3425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:46.692991 3425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:46.693079 3425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:46.693218 3425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:46.693282 3425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:46.726145 3425 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:45901
I20260808 03:57:46.726243 3626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:45901 every 8 connection(s)
I20260808 03:57:46.727624 3425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:57:46.728971 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 3425
I20260808 03:57:46.729243 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:57:46.730105 29679 external_mini_cluster.cc:1428] Setting key 8260d99396bf02147513723544c53da9
Aug 08 03:57:46 dist-test-slave-pww0 krb5kdc[3251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161466, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:46.750495 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:46.730879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:52627 (local address 127.28.251.254:36391)
0808 03:57:46.731344 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:46.731351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:46.732602 (+ 1251us) server_negotiation.cc:407] Connection header received
0808 03:57:46.734576 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:46.734582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:46.734666 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:46.734787 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:46.736478 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:46.737528 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:46.739178 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:46.739766 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:46.739893 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:46.743934 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:46.743940 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:46.743943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:46.743983 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:46.746718 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:46.747907 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:46.747912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:46.747914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:46.748057 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:46.748580 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:46.748584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:46.748586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:46.748881 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:46.748999 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:46.749993 (+ 994us) server_negotiation.cc:299] Negotiation successful
0808 03:57:46.750206 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":221,"threads_started":1}
I20260808 03:57:46.752177 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:46.731538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:36391 (local address 127.28.251.193:52627)
0808 03:57:46.732422 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:46.732477 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:46.734276 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:46.735017 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:46.735026 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:46.735638 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:46.736277 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:46.736297 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:46.737692 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:46.737696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:46.739006 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:46.739020 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:46.739994 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:46.740000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:46.740224 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:46.740920 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:46.740952 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:46.743645 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:46.746922 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:46.746934 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:46.746955 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:46.747756 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:46.748177 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:46.748182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:46.748185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:46.748461 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:46.749026 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:46.749036 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:46.749576 (+ 540us) client_negotiation.cc:769] Sending connection context
0808 03:57:46.749898 (+ 322us) client_negotiation.cc:240] Negotiation successful
0808 03:57:46.750505 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":352,"threads_started":1}
I20260808 03:57:46.753882 3628 heartbeater.cc:347] Connected to a master server at 127.28.251.254:36391
I20260808 03:57:46.754268 3628 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:46.755300 3628 heartbeater.cc:511] Master 127.28.251.254:36391 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
I20260808 03:57:46.757867 3321 ts_manager.cc:194] Registered new tserver with Master: 425a2b7fa02643fbba48795201649603 (127.28.251.193:45901)
I20260808 03:57:46.759387 3321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:52627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:57:46.788970 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36391
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:47.144642 3639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:47.144969 3639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:47.145128 3639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:47.152887 3639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:47.153026 3639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:47.153185 3639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:57:47.163797 3639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:47.165201 3639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:47.166657 3639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:47.180866 3649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:47.180912 3651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:47.180791 3647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:47.181830 3639 server_base.cc:1034] running on GCE node
I20260808 03:57:47.182510 3639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:47.183575 3639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:47.184796 3639 hybrid_clock.cc:648] HybridClock initialized: now 1786161467184739 us; error 49 us; skew 500 ppm
I20260808 03:57:47.185129 3639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:47 dist-test-slave-pww0 krb5kdc[3251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161467, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:47.188812 3639 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:57:47.190470 3639 webserver.cc:460] Webserver started at http://127.28.251.194:35203/ using document root <none> and password file <none>
I20260808 03:57:47.191244 3639 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:47.191356 3639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:47.191720 3639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:47.195006 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "9e20a8baad7f40dcb5347daed5d964a3"
format_stamp: "Formatted at 2026-08-08 03:57:47 on dist-test-slave-pww0"
server_key: "e462c406c8834a160b7d8111a911eec0"
server_key_iv: "9094fc6b297d771d43c57affddfc48b3"
server_key_version: "encryptionkey@0"
I20260808 03:57:47.196204 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "9e20a8baad7f40dcb5347daed5d964a3"
format_stamp: "Formatted at 2026-08-08 03:57:47 on dist-test-slave-pww0"
server_key: "e462c406c8834a160b7d8111a911eec0"
server_key_iv: "9094fc6b297d771d43c57affddfc48b3"
server_key_version: "encryptionkey@0"
I20260808 03:57:47.202077 3639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 03:57:47.206079 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:47.207645 3639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:47.207851 3639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "9e20a8baad7f40dcb5347daed5d964a3"
format_stamp: "Formatted at 2026-08-08 03:57:47 on dist-test-slave-pww0"
server_key: "e462c406c8834a160b7d8111a911eec0"
server_key_iv: "9094fc6b297d771d43c57affddfc48b3"
server_key_version: "encryptionkey@0"
I20260808 03:57:47.208069 3639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:47.233734 3639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:47.238704 3639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:47.239001 3639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:47.240134 3639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:47.241678 3639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:47.241761 3639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:47.241888 3639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:47.241948 3639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:47.274936 3639 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:39949
I20260808 03:57:47.274968 3832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:39949 every 8 connection(s)
I20260808 03:57:47.276482 3639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:57:47.286723 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 3639
I20260808 03:57:47.286965 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:57:47.287649 29679 external_mini_cluster.cc:1428] Setting key ce48ee2ce2a9603c2157ab3b833bc4ea
Aug 08 03:57:47 dist-test-slave-pww0 krb5kdc[3251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161467, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:47.300426 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:47.279143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:54547 (local address 127.28.251.254:36391)
0808 03:57:47.279716 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:47.279722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:47.280873 (+ 1151us) server_negotiation.cc:407] Connection header received
0808 03:57:47.282972 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:47.282980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:47.283077 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:47.283194 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:47.285062 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.285824 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.288437 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.289023 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.289136 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:47.293376 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:47.293383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:47.293387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:47.293432 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:47.296719 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:47.297697 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:47.297702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:47.297705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:47.297871 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:47.298371 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:47.298375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:47.298378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:47.298750 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:47.298907 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:47.299780 (+ 873us) server_negotiation.cc:299] Negotiation successful
0808 03:57:47.300031 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":240,"threads_started":1}
I20260808 03:57:47.301688 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:47.279763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:36391 (local address 127.28.251.194:54547)
0808 03:57:47.280650 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:47.280717 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:47.282638 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:47.283542 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:47.283551 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:47.284171 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:47.284844 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:47.284865 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.286023 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.286028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:47.288221 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:47.288242 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.289307 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.289316 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:47.289645 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:47.290387 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:47.290413 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:47.293043 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:47.296939 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:47.296949 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:47.296966 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:47.297527 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:47.297998 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:47.298002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:47.298005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:47.298248 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:47.298903 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:47.298910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:47.299313 (+ 403us) client_negotiation.cc:769] Sending connection context
0808 03:57:47.299645 (+ 332us) client_negotiation.cc:240] Negotiation successful
0808 03:57:47.300168 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":345,"threads_started":1}
I20260808 03:57:47.303282 3833 heartbeater.cc:347] Connected to a master server at 127.28.251.254:36391
I20260808 03:57:47.303676 3833 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:47.304688 3833 heartbeater.cc:511] Master 127.28.251.254:36391 requested a full tablet report, sending...
I20260808 03:57:47.306435 3321 ts_manager.cc:194] Registered new tserver with Master: 9e20a8baad7f40dcb5347daed5d964a3 (127.28.251.194:39949)
I20260808 03:57:47.307307 3321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:54547
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:57:47.347149 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36391
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:47.719043 3843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:47.719399 3843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:47.719574 3843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:47.727356 3843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:47.727531 3843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:47.727705 3843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:57:47.738340 3843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:47.739876 3843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:47.741537 3843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:47.755674 3854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:47.755651 3852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:47.755623 3851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:47.756465 3843 server_base.cc:1034] running on GCE node
I20260808 03:57:47.757171 3843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:47.758280 3843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:47.759557 3843 hybrid_clock.cc:648] HybridClock initialized: now 1786161467759482 us; error 74 us; skew 500 ppm
I20260808 03:57:47.759909 3843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:47 dist-test-slave-pww0 krb5kdc[3251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161467, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:47.763391 3628 heartbeater.cc:503] Master 127.28.251.254:36391 was elected leader, sending a full tablet report...
I20260808 03:57:47.763568 3843 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:57:47.765610 3843 webserver.cc:460] Webserver started at http://127.28.251.195:43953/ using document root <none> and password file <none>
I20260808 03:57:47.766412 3843 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:47.766532 3843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:47.766871 3843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:47.770177 3843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "2c6bed5ddbd34c38aaa5fc2481f3dbd5"
format_stamp: "Formatted at 2026-08-08 03:57:47 on dist-test-slave-pww0"
server_key: "c3508c5a33a39405e683bb5615cf0657"
server_key_iv: "0976525090c0b10c5597b9c6a363bc1c"
server_key_version: "encryptionkey@0"
I20260808 03:57:47.771106 3843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "2c6bed5ddbd34c38aaa5fc2481f3dbd5"
format_stamp: "Formatted at 2026-08-08 03:57:47 on dist-test-slave-pww0"
server_key: "c3508c5a33a39405e683bb5615cf0657"
server_key_iv: "0976525090c0b10c5597b9c6a363bc1c"
server_key_version: "encryptionkey@0"
I20260808 03:57:47.777088 3843 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260808 03:57:47.781052 3865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:47.782548 3843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:57:47.782719 3843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "2c6bed5ddbd34c38aaa5fc2481f3dbd5"
format_stamp: "Formatted at 2026-08-08 03:57:47 on dist-test-slave-pww0"
server_key: "c3508c5a33a39405e683bb5615cf0657"
server_key_iv: "0976525090c0b10c5597b9c6a363bc1c"
server_key_version: "encryptionkey@0"
I20260808 03:57:47.782967 3843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:47.813233 3843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:47.817916 3843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:47.818208 3843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:47.819289 3843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:47.820976 3843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:47.821077 3843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:47.821213 3843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:47.821275 3843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:47.854614 3843 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:35441
I20260808 03:57:47.854666 4057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:35441 every 8 connection(s)
I20260808 03:57:47.856021 3843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:57:47.866115 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 3843
I20260808 03:57:47.866350 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:57:47.866880 29679 external_mini_cluster.cc:1428] Setting key e97aa6701989be2fcca9917c3fe52c7d
Aug 08 03:57:47 dist-test-slave-pww0 krb5kdc[3251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161467, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:47.877094 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:47.858802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:40229 (local address 127.28.251.254:36391)
0808 03:57:47.859283 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:47.859287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:47.860241 (+ 954us) server_negotiation.cc:407] Connection header received
0808 03:57:47.862155 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:47.862161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:47.862247 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:47.862348 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:47.863993 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.864801 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.866762 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.867223 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.867333 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:47.871489 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:47.871496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:47.871502 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:47.871544 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:47.874157 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:47.874998 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:47.875002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:47.875004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:47.875126 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:47.875525 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:47.875528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:47.875530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:47.875810 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:47.875925 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:47.876633 (+ 708us) server_negotiation.cc:299] Negotiation successful
0808 03:57:47.876816 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":242,"threads_started":1}
I20260808 03:57:47.878147 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:47.859259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:36391 (local address 127.28.251.195:40229)
0808 03:57:47.860054 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:47.860099 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:47.861865 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:47.862603 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:47.862612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:47.863155 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:47.863809 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:47.863829 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.864963 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.864968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:47.866595 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:47.866611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.867500 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.867510 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:47.867738 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:47.868473 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:47.868497 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:47.871228 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:47.874328 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:47.874336 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:47.874353 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:47.874892 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:47.875205 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:47.875208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:47.875210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:47.875404 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:47.875928 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:47.875934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:47.876301 (+ 367us) client_negotiation.cc:769] Sending connection context
0808 03:57:47.876538 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 03:57:47.876978 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":334,"threads_started":1}
I20260808 03:57:47.879688 4058 heartbeater.cc:347] Connected to a master server at 127.28.251.254:36391
I20260808 03:57:47.880043 4058 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:47.880873 4058 heartbeater.cc:511] Master 127.28.251.254:36391 requested a full tablet report, sending...
I20260808 03:57:47.882470 3321 ts_manager.cc:194] Registered new tserver with Master: 2c6bed5ddbd34c38aaa5fc2481f3dbd5 (127.28.251.195:35441)
I20260808 03:57:47.883078 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 03:57:47.883363 3321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:40229
Aug 08 03:57:47 dist-test-slave-pww0 krb5kdc[3251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 03:57:47 dist-test-slave-pww0 krb5kdc[3251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:57:47.946344 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:47.930156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36288 (local address 127.28.251.193:45901)
0808 03:57:47.930858 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:47.930865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:47.930895 (+ 30us) server_negotiation.cc:407] Connection header received
0808 03:57:47.930963 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:47.930970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:47.931257 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:47.931429 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:47.933006 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.934058 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.935294 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.935939 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.936042 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:47.940213 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:47.940230 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:47.940249 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:47.940291 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:47.943269 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:47.944056 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:47.944065 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:47.944067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:47.944211 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:47.944637 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:47.944641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:47.944644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:47.945023 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:47.945368 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:47.945698 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 03:57:47.945987 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":371,"threads_started":1}
Aug 08 03:57:47 dist-test-slave-pww0 krb5kdc[3251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:47.965849 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:47.951408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.28.251.254:36391)
0808 03:57:47.951646 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:47.951653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:47.951852 (+ 199us) server_negotiation.cc:407] Connection header received
0808 03:57:47.952039 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:47.952043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:47.952135 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:47.952223 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:47.953545 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.954275 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.955575 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.955985 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.956080 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:47.960109 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:47.960115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:47.960118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:47.960161 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:47.962814 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:47.963651 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:47.963657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:47.963659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:47.963805 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:47.964228 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:47.964232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:47.964234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:47.964657 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:47.964918 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:47.965268 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 03:57:47.965523 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260808 03:57:47.966557 3321 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36742
I20260808 03:57:47.967768 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 3425
I20260808 03:57:47.982085 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:47.968739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.28.251.193:45901)
0808 03:57:47.968955 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:47.968961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:47.969225 (+ 264us) server_negotiation.cc:407] Connection header received
0808 03:57:47.969390 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:47.969396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:47.969485 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:47.969575 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:47.970898 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.971691 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.972961 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:47.973395 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:47.973519 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:47.976244 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:47.976252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:47.976257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:47.976305 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:47.979080 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:47.980042 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:47.980047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:47.980049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:47.980246 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:47.980680 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:47.980685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:47.980687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:47.981025 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:47.981232 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:47.981542 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 03:57:47.981756 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 03:57:47.982733 3621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:48.069371 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 3425
I20260808 03:57:48.095117 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 3639
Aug 08 03:57:48 dist-test-slave-pww0 krb5kdc[3251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 03:57:48.112196 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:48.096117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.28.251.194:39949)
0808 03:57:48.096680 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:48.096688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:48.096717 (+ 29us) server_negotiation.cc:407] Connection header received
0808 03:57:48.096785 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:48.096792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:48.097003 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:48.097152 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:48.098390 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:48.099506 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:48.100942 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:48.101753 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:48.101881 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:48.106147 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:48.106165 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:48.106182 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:48.106235 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:48.109278 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:48.109992 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:48.110000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:48.110002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:48.110144 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:48.110491 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:48.110494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:48.110496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:48.110875 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:48.111124 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:48.111503 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 03:57:48.111805 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":286,"threads_started":1}
I20260808 03:57:48.113564 3827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:48.198699 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 3639
I20260808 03:57:48.223701 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 3843
Aug 08 03:57:48 dist-test-slave-pww0 krb5kdc[3251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 03:57:48.241180 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:48.224638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.28.251.195:35441)
0808 03:57:48.225310 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:48.225317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:48.225351 (+ 34us) server_negotiation.cc:407] Connection header received
0808 03:57:48.225418 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:48.225426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:48.225646 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:48.225795 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:48.227297 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:48.228485 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:48.229847 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:48.230466 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:48.230572 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:48.235182 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:48.235199 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:48.235216 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:48.235269 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:48.238369 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:48.239144 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:48.239153 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:48.239155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:48.239293 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:48.239711 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:48.239714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:48.239716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:48.240116 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:48.240385 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:48.240613 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 03:57:48.240870 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":316,"threads_started":1}
I20260808 03:57:48.242575 4048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:48.326921 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 3843
I20260808 03:57:48.353434 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 3271
I20260808 03:57:48.354795 3374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:48.462952 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 3271
2026-08-08T03:57:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4390 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:57:48 dist-test-slave-pww0 krb5kdc[4090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:57:48 dist-test-slave-pww0 krb5kdc[4090](info): set up 2 sockets
Aug 08 03:57:48 dist-test-slave-pww0 krb5kdc[4090](info): commencing operation
krb5kdc: starting...
W20260808 03:57:49.728868 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:57:49 dist-test-slave-pww0 krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:57:49.991976 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:35697
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:35697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:50.343223 4111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:50.343636 4111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:50.343787 4111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:50.351402 4111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:50.351588 4111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:50.351681 4111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:50.351750 4111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:50.351814 4111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:57:50.362392 4111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:35697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:35697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:50.363899 4111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:50.365303 4111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:50.377393 4119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:50.377359 4123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:50.377321 4121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:50.378202 4111 server_base.cc:1034] running on GCE node
I20260808 03:57:50.379081 4111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:50.380681 4111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:50.381916 4111 hybrid_clock.cc:648] HybridClock initialized: now 1786161470381858 us; error 52 us; skew 500 ppm
I20260808 03:57:50.382314 4111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:50 dist-test-slave-pww0 krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161470, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:50.385547 4111 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:57:50.386763 4111 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:50.386899 4111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:50.387228 4111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:50.390380 4111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "dc1b8e3be9b1443bbfc1d27bfd524b0c"
format_stamp: "Formatted at 2026-08-08 03:57:50 on dist-test-slave-pww0"
server_key: "556b23932b8617145cc6fd4f989d0ab0"
server_key_iv: "9e43440385efb499c237b6f0de37edd9"
server_key_version: "encryptionkey@0"
I20260808 03:57:50.391212 4111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "dc1b8e3be9b1443bbfc1d27bfd524b0c"
format_stamp: "Formatted at 2026-08-08 03:57:50 on dist-test-slave-pww0"
server_key: "556b23932b8617145cc6fd4f989d0ab0"
server_key_iv: "9e43440385efb499c237b6f0de37edd9"
server_key_version: "encryptionkey@0"
I20260808 03:57:50.396813 4111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 03:57:50.400511 4130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:50.401906 4111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 03:57:50.402096 4111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "dc1b8e3be9b1443bbfc1d27bfd524b0c"
format_stamp: "Formatted at 2026-08-08 03:57:50 on dist-test-slave-pww0"
server_key: "556b23932b8617145cc6fd4f989d0ab0"
server_key_iv: "9e43440385efb499c237b6f0de37edd9"
server_key_version: "encryptionkey@0"
I20260808 03:57:50.402351 4111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:50.427625 4111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:50.432407 4111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:50.432678 4111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:50.450526 4111 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:35697
I20260808 03:57:50.450547 4210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:35697 every 8 connection(s)
I20260808 03:57:50.451651 4111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:50.455852 4211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:50.458597 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 4111
I20260808 03:57:50.458832 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:50.459522 29679 external_mini_cluster.cc:1428] Setting key 7f4109b901ac3d3e76ecd765b2b7209a
I20260808 03:57:50.469543 4211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c: Bootstrap starting.
I20260808 03:57:50.473480 4211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:50.474606 4211 log.cc:824] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c: Log is configured to *not* fsync() on all Append() calls
Aug 08 03:57:50 dist-test-slave-pww0 krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:50.477751 4211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c: No bootstrap required, opened a new log
I20260808 03:57:50.484566 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:50.461738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.28.251.254:35697)
0808 03:57:50.462658 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:50.462674 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:50.462729 (+ 55us) server_negotiation.cc:407] Connection header received
0808 03:57:50.465070 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:50.465104 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:50.465683 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:50.466167 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:50.467674 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:50.469687 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:50.471135 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:50.472264 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:50.472487 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:50.476695 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:50.476703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:50.476720 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:50.476766 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:50.480019 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:50.480840 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:50.480848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:50.480855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:50.481039 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:50.481446 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:50.481451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:50.481453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:50.482061 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:50.482393 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:50.482876 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 03:57:50.483361 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":387,"threads_started":1}
I20260808 03:57:50.486179 4211 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b8e3be9b1443bbfc1d27bfd524b0c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35697 } }
I20260808 03:57:50.486692 4211 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:50.486806 4211 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1b8e3be9b1443bbfc1d27bfd524b0c, State: Initialized, Role: FOLLOWER
I20260808 03:57:50.487529 4211 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b8e3be9b1443bbfc1d27bfd524b0c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35697 } }
I20260808 03:57:50.487812 4211 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:50.487986 4211 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:50.488147 4211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:50.490470 4211 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b8e3be9b1443bbfc1d27bfd524b0c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35697 } }
I20260808 03:57:50.491058 4211 leader_election.cc:302] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc1b8e3be9b1443bbfc1d27bfd524b0c; no voters:
I20260808 03:57:50.491731 4211 leader_election.cc:288] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:50.491889 4219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:50.492630 4219 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [term 1 LEADER]: Becoming Leader. State: Replica: dc1b8e3be9b1443bbfc1d27bfd524b0c, State: Running, Role: LEADER
I20260808 03:57:50.493232 4219 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b8e3be9b1443bbfc1d27bfd524b0c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35697 } }
I20260808 03:57:50.493974 4211 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:50.496470 4220 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc1b8e3be9b1443bbfc1d27bfd524b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b8e3be9b1443bbfc1d27bfd524b0c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35697 } } }
I20260808 03:57:50.496470 4221 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc1b8e3be9b1443bbfc1d27bfd524b0c. Latest consensus state: current_term: 1 leader_uuid: "dc1b8e3be9b1443bbfc1d27bfd524b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1b8e3be9b1443bbfc1d27bfd524b0c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 35697 } } }
I20260808 03:57:50.496906 4220 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:50.496908 4221 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:50.498660 4234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:50.504900 4234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:50.514438 4234 catalog_manager.cc:1347] Generated new cluster ID: d8de479722b84801870534d0ade6fd58
I20260808 03:57:50.514582 4234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:50.529167 4234 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:50.530650 4234 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:50.537454 4234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c: Generated new TSK 0
I20260808 03:57:50.538384 4234 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:57:50.611059 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35697
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:50.978919 4257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:50.979243 4257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:50.979360 4257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:50.987334 4257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:50.987526 4257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:50.987682 4257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:57:50.998294 4257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:50.999818 4257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:51.001263 4257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:51.015100 4269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:51.015126 4271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:51.015153 4273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:51.016043 4257 server_base.cc:1034] running on GCE node
I20260808 03:57:51.016736 4257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:51.017864 4257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:51.019098 4257 hybrid_clock.cc:648] HybridClock initialized: now 1786161471019040 us; error 51 us; skew 500 ppm
I20260808 03:57:51.019484 4257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:51 dist-test-slave-pww0 krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161471, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:51.027554 4257 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:57:51.028997 4257 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:51.029125 4257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:51.029471 4257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:51.032826 4257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "ed2d082b7e914d66a27b24aab267af2d"
format_stamp: "Formatted at 2026-08-08 03:57:51 on dist-test-slave-pww0"
server_key: "da374ae3aaaa0f944a6304bddabd13dc"
server_key_iv: "798249e9355a925e823d8c0de82471c8"
server_key_version: "encryptionkey@0"
I20260808 03:57:51.033721 4257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "ed2d082b7e914d66a27b24aab267af2d"
format_stamp: "Formatted at 2026-08-08 03:57:51 on dist-test-slave-pww0"
server_key: "da374ae3aaaa0f944a6304bddabd13dc"
server_key_iv: "798249e9355a925e823d8c0de82471c8"
server_key_version: "encryptionkey@0"
I20260808 03:57:51.039815 4257 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260808 03:57:51.043926 4282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:51.045482 4257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 03:57:51.045691 4257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "ed2d082b7e914d66a27b24aab267af2d"
format_stamp: "Formatted at 2026-08-08 03:57:51 on dist-test-slave-pww0"
server_key: "da374ae3aaaa0f944a6304bddabd13dc"
server_key_iv: "798249e9355a925e823d8c0de82471c8"
server_key_version: "encryptionkey@0"
I20260808 03:57:51.045949 4257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:51.073231 4257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:51.078042 4257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:51.078349 4257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:51.079432 4257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:51.080946 4257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:51.081048 4257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:51.081167 4257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:51.081241 4257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:51.114559 4257 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:37295
I20260808 03:57:51.114625 4456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:37295 every 8 connection(s)
I20260808 03:57:51.115500 4257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:57:51.123327 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 4257
I20260808 03:57:51.123562 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:57:51.124219 29679 external_mini_cluster.cc:1428] Setting key f01d60c9808025be60492e97f09739f6
Aug 08 03:57:51 dist-test-slave-pww0 krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161471, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:51.139094 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:51.118102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:51829 (local address 127.28.251.254:35697)
0808 03:57:51.118580 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:51.118585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:51.119654 (+ 1069us) server_negotiation.cc:407] Connection header received
0808 03:57:51.121554 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:51.121560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:51.121670 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:51.121775 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:51.123381 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:51.124212 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:51.126082 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:51.127571 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:51.127707 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:51.132477 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:51.132483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:51.132487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:51.132530 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:51.135731 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:51.136619 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:51.136624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:51.136627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:51.136785 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:51.137281 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:51.137286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:51.137288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:51.137672 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:51.137809 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:51.138629 (+ 820us) server_negotiation.cc:299] Negotiation successful
0808 03:57:51.138826 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
I20260808 03:57:51.140636 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:51.118684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:35697 (local address 127.28.251.193:51829)
0808 03:57:51.119479 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:51.119518 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:51.121259 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:51.122037 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:51.122046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:51.122596 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:51.123193 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:51.123211 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:51.124437 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:51.124443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:51.125899 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:51.125913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:51.127930 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:51.127941 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:51.128284 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:51.129105 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:51.129135 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:51.132150 (+ 3015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:51.135931 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:51.135940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:51.135955 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:51.136482 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:51.136900 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:51.136903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:51.136906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:51.137170 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:51.137818 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:51.137825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:51.138223 (+ 398us) client_negotiation.cc:769] Sending connection context
0808 03:57:51.138480 (+ 257us) client_negotiation.cc:240] Negotiation successful
0808 03:57:51.139111 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":308,"threads_started":1}
I20260808 03:57:51.142379 4457 heartbeater.cc:347] Connected to a master server at 127.28.251.254:35697
I20260808 03:57:51.142691 4457 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:51.143391 4457 heartbeater.cc:511] Master 127.28.251.254:35697 requested a full tablet report, sending...
I20260808 03:57:51.145915 4155 ts_manager.cc:194] Registered new tserver with Master: ed2d082b7e914d66a27b24aab267af2d (127.28.251.193:37295)
I20260808 03:57:51.147986 4155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:51829
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:57:51.188714 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35697
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:51.569162 4466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:51.569486 4466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:51.569625 4466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:51.578059 4466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:51.578197 4466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:51.578374 4466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:57:51.588819 4466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:51.590255 4466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:51.591801 4466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:51.605553 4473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:51.605700 4472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:51.605623 4476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:51.606753 4466 server_base.cc:1034] running on GCE node
I20260808 03:57:51.607499 4466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:51.608528 4466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:51.609788 4466 hybrid_clock.cc:648] HybridClock initialized: now 1786161471609732 us; error 37 us; skew 500 ppm
I20260808 03:57:51.610180 4466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:51 dist-test-slave-pww0 krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161471, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:51.613917 4466 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:57:51.615317 4466 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:51.615442 4466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:51.615863 4466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:51.619361 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "3371201426e8410d8155905fc87d8469"
format_stamp: "Formatted at 2026-08-08 03:57:51 on dist-test-slave-pww0"
server_key: "ebe0cec85d69f392e5ad4f21a4677f06"
server_key_iv: "641aeb4f5d040bba9c794aefddaf61dc"
server_key_version: "encryptionkey@0"
I20260808 03:57:51.620415 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "3371201426e8410d8155905fc87d8469"
format_stamp: "Formatted at 2026-08-08 03:57:51 on dist-test-slave-pww0"
server_key: "ebe0cec85d69f392e5ad4f21a4677f06"
server_key_iv: "641aeb4f5d040bba9c794aefddaf61dc"
server_key_version: "encryptionkey@0"
I20260808 03:57:51.626541 4466 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260808 03:57:51.630777 4484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:51.632314 4466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:51.632494 4466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "3371201426e8410d8155905fc87d8469"
format_stamp: "Formatted at 2026-08-08 03:57:51 on dist-test-slave-pww0"
server_key: "ebe0cec85d69f392e5ad4f21a4677f06"
server_key_iv: "641aeb4f5d040bba9c794aefddaf61dc"
server_key_version: "encryptionkey@0"
I20260808 03:57:51.632762 4466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:51.660768 4466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:51.665230 4466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:51.665517 4466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:51.666587 4466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:51.668139 4466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:51.668234 4466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:51.668344 4466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:51.668412 4466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:51.701020 4466 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:43625
I20260808 03:57:51.701072 4663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:43625 every 8 connection(s)
I20260808 03:57:51.701917 4466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:57:51.706919 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 4466
I20260808 03:57:51.707122 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:57:51.707621 29679 external_mini_cluster.cc:1428] Setting key c1cae4e27743d9b8cf87650b8e4d552c
Aug 08 03:57:51 dist-test-slave-pww0 krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161471, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:51.723510 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:51.704426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:48673 (local address 127.28.251.254:35697)
0808 03:57:51.704900 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:51.704905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:51.705789 (+ 884us) server_negotiation.cc:407] Connection header received
0808 03:57:51.707882 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:51.707888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:51.707976 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:51.708076 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:51.709694 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:51.710443 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:51.712137 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:51.712579 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:51.712698 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:51.716658 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:51.716665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:51.716669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:51.716712 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:51.719675 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:51.720755 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:51.720758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:51.720761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:51.720942 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:51.721401 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:51.721406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:51.721408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:51.721784 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:51.721948 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:51.722801 (+ 853us) server_negotiation.cc:299] Negotiation successful
0808 03:57:51.723075 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":255,"threads_started":1}
I20260808 03:57:51.724474 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:51.704882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:35697 (local address 127.28.251.194:48673)
0808 03:57:51.705607 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:51.705651 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:51.707576 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:51.708362 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:51.708370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:51.708911 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:51.709523 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:51.709542 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:51.710605 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:51.710610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:51.711973 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:51.711988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:51.712836 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:51.712844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:51.713094 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:51.713775 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:51.713801 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:51.716348 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:51.719874 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:51.719886 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:51.719910 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:51.720629 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:51.721059 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:51.721063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:51.721065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:51.721277 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:51.721960 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:51.721967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:51.722389 (+ 422us) client_negotiation.cc:769] Sending connection context
0808 03:57:51.722657 (+ 268us) client_negotiation.cc:240] Negotiation successful
0808 03:57:51.723123 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":289,"threads_started":1}
I20260808 03:57:51.726222 4664 heartbeater.cc:347] Connected to a master server at 127.28.251.254:35697
I20260808 03:57:51.726629 4664 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:51.727567 4664 heartbeater.cc:511] Master 127.28.251.254:35697 requested a full tablet report, sending...
I20260808 03:57:51.729444 4155 ts_manager.cc:194] Registered new tserver with Master: 3371201426e8410d8155905fc87d8469 (127.28.251.194:43625)
I20260808 03:57:51.730366 4155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:48673
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:57:51.766556 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35697
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:57:52.126765 4676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:52.127106 4676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:52.127270 4676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:52.135603 4676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:52.135780 4676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:52.135946 4676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:57:52.146603 4676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:35697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:52.148152 4676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:52.149642 4676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 03:57:52.152421 4457 heartbeater.cc:503] Master 127.28.251.254:35697 was elected leader, sending a full tablet report...
W20260808 03:57:52.163691 4686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:52.163944 4688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:52.164207 4685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:52.164691 4676 server_base.cc:1034] running on GCE node
I20260808 03:57:52.165370 4676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:52.166409 4676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:52.167626 4676 hybrid_clock.cc:648] HybridClock initialized: now 1786161472167570 us; error 54 us; skew 500 ppm
I20260808 03:57:52.167973 4676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:57:52 dist-test-slave-pww0 krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161472, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:57:52.171399 4676 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:57:52.172748 4676 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:52.172868 4676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:52.173202 4676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:52.176479 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "e4bdaaf626f04c5ab6e792d695134ed2"
format_stamp: "Formatted at 2026-08-08 03:57:52 on dist-test-slave-pww0"
server_key: "ad758770e6d02495736795bc6cd10092"
server_key_iv: "c9157a1b4d1b688d68e658e0ddb66d04"
server_key_version: "encryptionkey@0"
I20260808 03:57:52.177388 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "e4bdaaf626f04c5ab6e792d695134ed2"
format_stamp: "Formatted at 2026-08-08 03:57:52 on dist-test-slave-pww0"
server_key: "ad758770e6d02495736795bc6cd10092"
server_key_iv: "c9157a1b4d1b688d68e658e0ddb66d04"
server_key_version: "encryptionkey@0"
I20260808 03:57:52.183095 4676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 03:57:52.186950 4700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:52.188493 4676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 03:57:52.188684 4676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "e4bdaaf626f04c5ab6e792d695134ed2"
format_stamp: "Formatted at 2026-08-08 03:57:52 on dist-test-slave-pww0"
server_key: "ad758770e6d02495736795bc6cd10092"
server_key_iv: "c9157a1b4d1b688d68e658e0ddb66d04"
server_key_version: "encryptionkey@0"
I20260808 03:57:52.188923 4676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:52.204334 4676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:52.209059 4676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:52.209354 4676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:52.210424 4676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:52.211970 4676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:52.212069 4676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:52.212178 4676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:52.212245 4676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:52.243453 4676 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:46489
I20260808 03:57:52.243501 4888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:46489 every 8 connection(s)
I20260808 03:57:52.244439 4676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:57:52.254015 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 4676
I20260808 03:57:52.254217 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:57:52.254673 29679 external_mini_cluster.cc:1428] Setting key 875fad5accfa0ebf594dbf9646fb2ab8
Aug 08 03:57:52 dist-test-slave-pww0 krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161472, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:57:52.266599 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.247058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:57653 (local address 127.28.251.254:35697)
0808 03:57:52.247582 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.247586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.248506 (+ 920us) server_negotiation.cc:407] Connection header received
0808 03:57:52.250563 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.250570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.250658 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.250770 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:52.252597 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.253442 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.255352 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.255986 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.256138 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.260331 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:52.260337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:52.260340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:52.260381 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:52.263052 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:52.264001 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:52.264005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:52.264008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:52.264139 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:52.264614 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:52.264618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:52.264620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:52.264942 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:52.265083 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.266065 (+ 982us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.266308 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":248,"threads_started":1}
I20260808 03:57:52.267627 4895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.247579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:35697 (local address 127.28.251.195:57653)
0808 03:57:52.248322 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:52.248370 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:52.250217 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:52.251060 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:52.251069 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:52.251699 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:52.252389 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.252411 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.253635 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.253639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.255131 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.255149 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.256281 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.256290 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.256524 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.257282 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:57:52.257316 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:52.260036 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:52.263233 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:52.263242 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:52.263258 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:52.263872 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:52.264241 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:57:52.264245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:57:52.264247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:57:52.264455 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:57:52.265127 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:52.265134 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:52.265613 (+ 479us) client_negotiation.cc:769] Sending connection context
0808 03:57:52.265919 (+ 306us) client_negotiation.cc:240] Negotiation successful
0808 03:57:52.266376 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":291,"threads_started":1}
I20260808 03:57:52.269340 4891 heartbeater.cc:347] Connected to a master server at 127.28.251.254:35697
I20260808 03:57:52.269714 4891 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:52.270491 4891 heartbeater.cc:511] Master 127.28.251.254:35697 requested a full tablet report, sending...
I20260808 03:57:52.272156 4155 ts_manager.cc:194] Registered new tserver with Master: e4bdaaf626f04c5ab6e792d695134ed2 (127.28.251.195:46489)
I20260808 03:57:52.273012 4155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:57653
I20260808 03:57:52.283507 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 03:57:52.301842 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.288507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.28.251.254:35697)
0808 03:57:52.288773 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.288780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.289133 (+ 353us) server_negotiation.cc:407] Connection header received
0808 03:57:52.289309 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.289313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.289404 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.289494 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:52.291028 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.291880 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.293328 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.293903 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.294027 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.296564 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:52.296569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:52.296573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:52.296615 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:52.299318 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:52.300034 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:52.300038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:52.300041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:52.300188 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:52.300510 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:52.300513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:52.300515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:52.300869 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:52.301063 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.301328 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.301524 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 03:57:52.306499 4155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:57:52.313796 4155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:57:52.353345 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.327495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:37295 (local address 127.0.0.1:34136)
0808 03:57:52.343571 (+ 16076us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:52.343593 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:52.343793 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:52.344980 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:52.344988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:52.345024 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:52.345480 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.345497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.347680 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.347684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.350460 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.350475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.352425 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.352431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.352601 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.352641 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 03:57:52.352732 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 03:57:52.352911 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15897,"mutex_wait_us":96,"thread_start_us":14493,"threads_started":1}
I20260808 03:57:52.354652 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.327923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34136 (local address 127.28.251.193:37295)
0808 03:57:52.341960 (+ 14037us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.341969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.343972 (+ 2003us) server_negotiation.cc:407] Connection header received
0808 03:57:52.344130 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.344141 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.344481 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.345504 (+ 1023us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:52.345808 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.347527 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.350624 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.352178 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.353827 (+ 1649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.354023 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.354106 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.354312 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13843,"thread_start_us":13692,"threads_started":1}
I20260808 03:57:52.356228 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.345324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:46489 (local address 127.0.0.1:42358)
0808 03:57:52.346493 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:52.346513 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:52.346657 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:52.347637 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:52.347642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:52.347671 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:52.348206 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.348218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.350477 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.350482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.352920 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.352934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.355234 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.355241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.355494 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.355531 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 03:57:52.355607 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 03:57:52.355745 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":2,"thread_start_us":880,"threads_started":1}
I20260808 03:57:52.356323 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.327308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:43625 (local address 127.0.0.1:38168)
0808 03:57:52.340772 (+ 13464us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:52.340841 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:52.341145 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:52.346767 (+ 5622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:52.346771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:52.346801 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:52.347208 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.347223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.349257 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.349261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.351203 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.351220 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.355152 (+ 3932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.355161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.355389 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.355631 (+ 242us) client_negotiation.cc:769] Sending connection context
0808 03:57:52.355729 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 03:57:52.355871 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13069,"mutex_wait_us":205,"thread_start_us":15885,"threads_started":1}
I20260808 03:57:52.357061 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.327524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.28.251.194:43625)
0808 03:57:52.346148 (+ 18624us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.346159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.346192 (+ 33us) server_negotiation.cc:407] Connection header received
0808 03:57:52.346296 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.346304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.346555 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.347531 (+ 976us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:52.347831 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.349117 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.351427 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.352929 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.353110 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.353270 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.356489 (+ 3219us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.356722 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18415,"thread_start_us":18298,"threads_started":1}
I20260808 03:57:52.357792 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.343383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.28.251.195:46489)
0808 03:57:52.345320 (+ 1937us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.345331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.346643 (+ 1312us) server_negotiation.cc:407] Connection header received
0808 03:57:52.346881 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.346889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.347129 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.347309 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:52.348589 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.350284 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.353114 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.354841 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.355220 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.357131 (+ 1911us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.357235 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.357429 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1714,"thread_start_us":1521,"threads_started":1}
I20260808 03:57:52.359225 4355 tablet_service.cc:1467] Processing CreateTablet for tablet 8e96dccd81b241b487b2cbec19f80027 (DEFAULT_TABLE table=test-table [id=4dcb44db402746c28bc53249f5c26ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:52.360805 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e96dccd81b241b487b2cbec19f80027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:52.360463 4561 tablet_service.cc:1467] Processing CreateTablet for tablet 8e96dccd81b241b487b2cbec19f80027 (DEFAULT_TABLE table=test-table [id=4dcb44db402746c28bc53249f5c26ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:52.362265 4561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e96dccd81b241b487b2cbec19f80027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:52.363788 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 8e96dccd81b241b487b2cbec19f80027 (DEFAULT_TABLE table=test-table [id=4dcb44db402746c28bc53249f5c26ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:52.365362 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e96dccd81b241b487b2cbec19f80027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:52.376292 4926 tablet_bootstrap.cc:492] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: Bootstrap starting.
I20260808 03:57:52.379710 4928 tablet_bootstrap.cc:492] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: Bootstrap starting.
I20260808 03:57:52.380641 4926 tablet_bootstrap.cc:654] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:52.381677 4929 tablet_bootstrap.cc:492] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: Bootstrap starting.
I20260808 03:57:52.382125 4926 log.cc:824] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:52.383499 4928 tablet_bootstrap.cc:654] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:52.384994 4928 log.cc:824] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:52.385449 4929 tablet_bootstrap.cc:654] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:52.386251 4926 tablet_bootstrap.cc:492] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: No bootstrap required, opened a new log
I20260808 03:57:52.386763 4926 ts_tablet_manager.cc:1397] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260808 03:57:52.386775 4929 log.cc:824] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:52.390045 4928 tablet_bootstrap.cc:492] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: No bootstrap required, opened a new log
I20260808 03:57:52.390244 4929 tablet_bootstrap.cc:492] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: No bootstrap required, opened a new log
I20260808 03:57:52.390391 4928 ts_tablet_manager.cc:1397] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260808 03:57:52.390553 4929 ts_tablet_manager.cc:1397] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260808 03:57:52.397897 4926 raft_consensus.cc:348] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3371201426e8410d8155905fc87d8469" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43625 } } peers { permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 } } peers { permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 } }
I20260808 03:57:52.398453 4926 raft_consensus.cc:374] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:52.398617 4926 raft_consensus.cc:729] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3371201426e8410d8155905fc87d8469, State: Initialized, Role: FOLLOWER
I20260808 03:57:52.398933 4928 raft_consensus.cc:348] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3371201426e8410d8155905fc87d8469" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43625 } } peers { permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 } } peers { permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 } }
I20260808 03:57:52.399387 4928 raft_consensus.cc:374] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:52.399552 4928 raft_consensus.cc:729] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2d082b7e914d66a27b24aab267af2d, State: Initialized, Role: FOLLOWER
I20260808 03:57:52.399338 4926 consensus_queue.cc:260] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3371201426e8410d8155905fc87d8469" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43625 } } peers { permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 } } peers { permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 } }
I20260808 03:57:52.400172 4928 consensus_queue.cc:260] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3371201426e8410d8155905fc87d8469" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43625 } } peers { permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 } } peers { permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 } }
I20260808 03:57:52.401677 4929 raft_consensus.cc:348] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3371201426e8410d8155905fc87d8469" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43625 } } peers { permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 } } peers { permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 } }
I20260808 03:57:52.402045 4928 ts_tablet_manager.cc:1428] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260808 03:57:52.402266 4929 raft_consensus.cc:374] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:52.402379 4929 raft_consensus.cc:729] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bdaaf626f04c5ab6e792d695134ed2, State: Initialized, Role: FOLLOWER
I20260808 03:57:52.403172 4929 consensus_queue.cc:260] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3371201426e8410d8155905fc87d8469" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43625 } } peers { permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 } } peers { permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 } }
I20260808 03:57:52.405194 4929 ts_tablet_manager.cc:1428] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260808 03:57:52.405449 4891 heartbeater.cc:503] Master 127.28.251.254:35697 was elected leader, sending a full tablet report...
I20260808 03:57:52.406385 4664 heartbeater.cc:503] Master 127.28.251.254:35697 was elected leader, sending a full tablet report...
I20260808 03:57:52.407119 4926 ts_tablet_manager.cc:1428] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: Time spent starting tablet: real 0.020s user 0.010s sys 0.010s
W20260808 03:57:52.453996 4666 tablet.cc:2369] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 03:57:52.495997 4892 tablet.cc:2369] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:52.542184 4936 raft_consensus.cc:482] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:52.542526 4936 raft_consensus.cc:504] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3371201426e8410d8155905fc87d8469" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43625 } } peers { permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 } } peers { permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 } }
I20260808 03:57:52.544498 4936 leader_election.cc:288] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed2d082b7e914d66a27b24aab267af2d (127.28.251.193:37295), e4bdaaf626f04c5ab6e792d695134ed2 (127.28.251.195:46489)
I20260808 03:57:52.552408 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.546252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:49009 (local address 127.28.251.195:46489)
0808 03:57:52.546471 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.546477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.546679 (+ 202us) server_negotiation.cc:407] Connection header received
0808 03:57:52.546837 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.546843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.546919 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.547021 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:52.547672 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.548448 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.550522 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.551377 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.551572 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.551653 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.552024 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.552157 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 03:57:52.552399 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.544973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:41335 (local address 127.28.251.193:37295)
0808 03:57:52.545226 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.545233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.545930 (+ 697us) server_negotiation.cc:407] Connection header received
0808 03:57:52.546440 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.546447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.546540 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.546597 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:52.547533 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.548324 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.550475 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.551353 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.551541 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.551626 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.551987 (+ 361us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.552126 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 03:57:52.552584 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.545840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:46489 (local address 127.28.251.194:49009)
0808 03:57:52.546567 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:52.546583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:52.546708 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:52.547062 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:52.547065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:52.547089 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:52.547517 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.547530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.548614 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.548618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.550302 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.550319 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.551689 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.551694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.551852 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.551878 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 03:57:52.551974 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 03:57:52.552115 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":35,"thread_start_us":323,"threads_started":1}
I20260808 03:57:52.552620 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.544916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:37295 (local address 127.28.251.194:41335)
0808 03:57:52.545691 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:52.545743 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:52.546185 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:52.546820 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:52.546825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:52.546857 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:52.547328 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.547342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.548493 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.548499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.550273 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.550289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.551644 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.551652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.551816 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.551845 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 03:57:52.551925 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 03:57:52.552066 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":3,"thread_start_us":451,"threads_started":1}
I20260808 03:57:52.553969 4386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e96dccd81b241b487b2cbec19f80027" candidate_uuid: "3371201426e8410d8155905fc87d8469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2d082b7e914d66a27b24aab267af2d" is_pre_election: true
I20260808 03:57:52.554054 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e96dccd81b241b487b2cbec19f80027" candidate_uuid: "3371201426e8410d8155905fc87d8469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" is_pre_election: true
I20260808 03:57:52.554584 4386 raft_consensus.cc:2454] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3371201426e8410d8155905fc87d8469 in term 0.
I20260808 03:57:52.554579 4812 raft_consensus.cc:2454] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3371201426e8410d8155905fc87d8469 in term 0.
I20260808 03:57:52.554845 4940 raft_consensus.cc:482] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:52.555078 4940 raft_consensus.cc:504] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3371201426e8410d8155905fc87d8469" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43625 } } peers { permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 } } peers { permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 } }
I20260808 03:57:52.555423 4491 leader_election.cc:302] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3371201426e8410d8155905fc87d8469, ed2d082b7e914d66a27b24aab267af2d; no voters:
I20260808 03:57:52.555927 4936 raft_consensus.cc:2781] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:57:52.556087 4936 raft_consensus.cc:482] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:57:52.556217 4936 raft_consensus.cc:3037] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:52.557016 4940 leader_election.cc:288] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3371201426e8410d8155905fc87d8469 (127.28.251.194:43625), e4bdaaf626f04c5ab6e792d695134ed2 (127.28.251.195:46489)
I20260808 03:57:52.559224 4936 raft_consensus.cc:504] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3371201426e8410d8155905fc87d8469" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43625 } } peers { permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 } } peers { permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 } }
I20260808 03:57:52.560012 4936 leader_election.cc:288] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [CANDIDATE]: Term 1 election: Requested vote from peers ed2d082b7e914d66a27b24aab267af2d (127.28.251.193:37295), e4bdaaf626f04c5ab6e792d695134ed2 (127.28.251.195:46489)
I20260808 03:57:52.560652 4386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e96dccd81b241b487b2cbec19f80027" candidate_uuid: "3371201426e8410d8155905fc87d8469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2d082b7e914d66a27b24aab267af2d"
I20260808 03:57:52.560701 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e96dccd81b241b487b2cbec19f80027" candidate_uuid: "3371201426e8410d8155905fc87d8469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bdaaf626f04c5ab6e792d695134ed2"
I20260808 03:57:52.560928 4386 raft_consensus.cc:3037] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:52.560949 4812 raft_consensus.cc:3037] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:52.564445 4812 raft_consensus.cc:2454] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3371201426e8410d8155905fc87d8469 in term 1.
I20260808 03:57:52.564622 4386 raft_consensus.cc:2454] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3371201426e8410d8155905fc87d8469 in term 1.
I20260808 03:57:52.565048 4491 leader_election.cc:302] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3371201426e8410d8155905fc87d8469, e4bdaaf626f04c5ab6e792d695134ed2; no voters:
I20260808 03:57:52.566067 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.557935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:56583 (local address 127.28.251.195:46489)
0808 03:57:52.558133 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.558138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.558161 (+ 23us) server_negotiation.cc:407] Connection header received
0808 03:57:52.558443 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.558450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.558547 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.558618 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:52.559479 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.560605 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.563511 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.564820 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.564994 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.565074 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.565746 (+ 672us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.565859 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 03:57:52.566079 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.557380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:46489 (local address 127.28.251.193:56583)
0808 03:57:52.558056 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:52.558076 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:52.558240 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:52.558820 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:52.558825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:52.558856 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:52.559282 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.559296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.560791 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.560797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.563248 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.563264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.565227 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.565234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.565420 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.565459 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 03:57:52.565561 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 03:57:52.565714 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":779,"threads_started":1}
I20260808 03:57:52.565450 4936 raft_consensus.cc:2781] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:52.567018 4812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e96dccd81b241b487b2cbec19f80027" candidate_uuid: "ed2d082b7e914d66a27b24aab267af2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" is_pre_election: true
I20260808 03:57:52.567520 4812 raft_consensus.cc:2379] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ed2d082b7e914d66a27b24aab267af2d in current term 1: Already voted for candidate 3371201426e8410d8155905fc87d8469 in this term.
I20260808 03:57:52.567679 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.557948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:58537 (local address 127.28.251.194:43625)
0808 03:57:52.558143 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.558148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.560079 (+ 1931us) server_negotiation.cc:407] Connection header received
0808 03:57:52.560350 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.560356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.560433 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.560501 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:57:52.561479 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.562526 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.565095 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.566422 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.566580 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.566655 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.567233 (+ 578us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.567372 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 03:57:52.567730 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.557425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:43625 (local address 127.28.251.193:58537)
0808 03:57:52.559975 (+ 2550us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:52.560000 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:52.560185 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:52.560714 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:52.560720 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:52.560746 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:57:52.561250 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.561270 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.562731 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.562736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.564899 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:52.564916 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.566895 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.566906 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:52.567045 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.567070 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 03:57:52.567182 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 03:57:52.567332 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2424,"thread_start_us":2269,"threads_started":1}
I20260808 03:57:52.568351 4936 raft_consensus.cc:686] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 1 LEADER]: Becoming Leader. State: Replica: 3371201426e8410d8155905fc87d8469, State: Running, Role: LEADER
I20260808 03:57:52.568802 4591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e96dccd81b241b487b2cbec19f80027" candidate_uuid: "ed2d082b7e914d66a27b24aab267af2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3371201426e8410d8155905fc87d8469" is_pre_election: true
I20260808 03:57:52.569032 4936 consensus_queue.cc:237] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3371201426e8410d8155905fc87d8469" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43625 } } peers { permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 } } peers { permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 } }
I20260808 03:57:52.569931 4289 leader_election.cc:302] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed2d082b7e914d66a27b24aab267af2d; no voters: 3371201426e8410d8155905fc87d8469, e4bdaaf626f04c5ab6e792d695134ed2
I20260808 03:57:52.570380 4940 raft_consensus.cc:2726] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 03:57:52.574177 4153 catalog_manager.cc:5495] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3371201426e8410d8155905fc87d8469 (127.28.251.194). New cstate: current_term: 1 leader_uuid: "3371201426e8410d8155905fc87d8469" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3371201426e8410d8155905fc87d8469" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 } health_report { overall_health: UNKNOWN } } }
I20260808 03:57:52.598131 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.592143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38184 (local address 127.28.251.194:43625)
0808 03:57:52.592331 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.592336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.592470 (+ 134us) server_negotiation.cc:407] Connection header received
0808 03:57:52.592708 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.592712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.592824 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.592961 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:57:52.593384 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.594248 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.595746 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.596205 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.596313 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.596895 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:57:52.597451 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:57:52.597580 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.597749 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.597829 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 03:57:52.607024 4812 raft_consensus.cc:1261] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2 [term 1 FOLLOWER]: Refusing update from remote peer 3371201426e8410d8155905fc87d8469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:52.607056 4386 raft_consensus.cc:1261] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 1 FOLLOWER]: Refusing update from remote peer 3371201426e8410d8155905fc87d8469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:52.608057 4949 consensus_queue.cc:1035] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4bdaaf626f04c5ab6e792d695134ed2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 46489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:52.608471 4936 consensus_queue.cc:1035] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2d082b7e914d66a27b24aab267af2d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 37295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 03:57:52.618148 4458 tablet.cc:2369] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:52.620954 4957 mvcc.cc:205] Tried to move back new op lower bound from 7316117391787405312 to 7316117391647158272. Current Snapshot: MvccSnapshot[applied={T|T < 7316117391787405312}]
I20260808 03:57:52.622433 4962 mvcc.cc:205] Tried to move back new op lower bound from 7316117391787405312 to 7316117391647158272. Current Snapshot: MvccSnapshot[applied={T|T < 7316117391787405312}]
I20260808 03:57:52.627364 4958 mvcc.cc:205] Tried to move back new op lower bound from 7316117391787405312 to 7316117391647158272. Current Snapshot: MvccSnapshot[applied={T|T < 7316117391787405312}]
I20260808 03:57:52.644171 4154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:52.644482 4154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:52.648494 4154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc1b8e3be9b1443bbfc1d27bfd524b0c: Sending DeleteTablet for 3 replicas of tablet 8e96dccd81b241b487b2cbec19f80027
I20260808 03:57:52.649797 4561 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e96dccd81b241b487b2cbec19f80027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:52 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:38168
I20260808 03:57:52.649964 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e96dccd81b241b487b2cbec19f80027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:52 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:42358
I20260808 03:57:52.649979 4355 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e96dccd81b241b487b2cbec19f80027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:52 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:34136
I20260808 03:57:52.652637 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 4257
I20260808 03:57:52.660537 4982 tablet_replica.cc:330] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: stopping tablet replica
I20260808 03:57:52.661289 4982 raft_consensus.cc:2229] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:52.662101 4980 tablet_replica.cc:330] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: stopping tablet replica
I20260808 03:57:52.662768 4982 raft_consensus.cc:2258] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:52.663319 4980 raft_consensus.cc:2229] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:57:52.664477 4980 raft_consensus.cc:2258] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:52.668946 4980 ts_tablet_manager.cc:1905] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:52.667389 4982 ts_tablet_manager.cc:1905] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:52.677702 4982 ts_tablet_manager.cc:1918] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:57:52.678081 4982 log.cc:1196] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/8e96dccd81b241b487b2cbec19f80027
I20260808 03:57:52.678946 4982 ts_tablet_manager.cc:1939] T 8e96dccd81b241b487b2cbec19f80027 P ed2d082b7e914d66a27b24aab267af2d: Deleting consensus metadata
I20260808 03:57:52.681525 4137 catalog_manager.cc:4841] TS ed2d082b7e914d66a27b24aab267af2d (127.28.251.193:37295): tablet 8e96dccd81b241b487b2cbec19f80027 (table test-table [id=4dcb44db402746c28bc53249f5c26ae7]) successfully deleted
I20260808 03:57:52.682806 4980 ts_tablet_manager.cc:1918] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:57:52.683018 4981 tablet_replica.cc:330] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: stopping tablet replica
Aug 08 03:57:52 dist-test-slave-pww0 krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:57:52.683113 4980 log.cc:1196] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/8e96dccd81b241b487b2cbec19f80027
I20260808 03:57:52.683634 4981 raft_consensus.cc:2229] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:52.683863 4980 ts_tablet_manager.cc:1939] T 8e96dccd81b241b487b2cbec19f80027 P 3371201426e8410d8155905fc87d8469: Deleting consensus metadata
I20260808 03:57:52.684355 4981 raft_consensus.cc:2258] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:52.685945 4136 catalog_manager.cc:4841] TS 3371201426e8410d8155905fc87d8469 (127.28.251.194:43625): tablet 8e96dccd81b241b487b2cbec19f80027 (table test-table [id=4dcb44db402746c28bc53249f5c26ae7]) successfully deleted
I20260808 03:57:52.687196 4981 ts_tablet_manager.cc:1905] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:52.692569 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.654558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.28.251.193:37295)
0808 03:57:52.654779 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.654787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.658492 (+ 3705us) server_negotiation.cc:407] Connection header received
0808 03:57:52.658687 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.658694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.658809 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.658926 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:52.663823 (+ 4897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.665140 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.668481 (+ 3341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.669218 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.669335 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.684639 (+ 15304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:52.684658 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:52.684679 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:52.684738 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:52.688825 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:52.689912 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:52.689924 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:52.689927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:52.690133 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:52.690669 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:52.690674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:52.690677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:52.691170 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:52.691541 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.691934 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.692180 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 03:57:52.693399 4450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:52.694017 4981 ts_tablet_manager.cc:1918] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:57:52.694435 4981 log.cc:1196] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/8e96dccd81b241b487b2cbec19f80027
I20260808 03:57:52.695729 4981 ts_tablet_manager.cc:1939] T 8e96dccd81b241b487b2cbec19f80027 P e4bdaaf626f04c5ab6e792d695134ed2: Deleting consensus metadata
I20260808 03:57:52.698277 4137 catalog_manager.cc:4841] TS e4bdaaf626f04c5ab6e792d695134ed2 (127.28.251.195:46489): tablet 8e96dccd81b241b487b2cbec19f80027 (table test-table [id=4dcb44db402746c28bc53249f5c26ae7]) successfully deleted
I20260808 03:57:52.832782 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 4257
I20260808 03:57:52.875692 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 4466
Aug 08 03:57:52 dist-test-slave-pww0 krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 03:57:52.892207 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:52.876508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.28.251.194:43625)
0808 03:57:52.876768 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:52.876774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:52.876795 (+ 21us) server_negotiation.cc:407] Connection header received
0808 03:57:52.876924 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:52.876930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:52.876999 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:52.877090 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:52.878380 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.879160 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.880472 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:52.880955 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:52.881084 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:52.885735 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:52.885752 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:52.885770 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:52.885817 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:52.889064 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:52.890017 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:52.890025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:52.890028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:52.890190 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:52.890653 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:52.890656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:52.890658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:52.891077 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:52.891334 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:52.891720 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 03:57:52.891936 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260808 03:57:52.893060 4656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:53.013208 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 4466
I20260808 03:57:53.039798 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 4676
Aug 08 03:57:53 dist-test-slave-pww0 krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 03:57:53.054761 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:53.040582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.28.251.195:46489)
0808 03:57:53.040790 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:53.040795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:53.040815 (+ 20us) server_negotiation.cc:407] Connection header received
0808 03:57:53.040959 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:53.040963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:53.041048 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:53.041161 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:53.042502 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:53.043201 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:53.044450 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:53.044889 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:53.044988 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:53.049326 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:53.049348 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:53.049365 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:57:53.049407 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:53.052311 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:53.052951 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:53.052960 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:53.052962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:53.053103 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:57:53.053395 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:57:53.053398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:57:53.053400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:57:53.053795 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:53.054045 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:53.054249 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 03:57:53.054427 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 03:57:53.055483 4880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:53.149546 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 4676
I20260808 03:57:53.175899 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 4111
I20260808 03:57:53.177227 4205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:53.291889 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 4111
2026-08-08T03:57:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4825 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T03:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:53Z Disabled control of system clock
I20260808 03:57:53.382632 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:33035
--webserver_interface=127.28.251.254
--webserver_port=0
--builtin_ntp_servers=127.28.251.212:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 03:57:53.733341 4996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:53.733681 4996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:53.733814 4996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:53.741307 4996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:53.741441 4996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:53.741508 4996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:53.741590 4996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:53.741648 4996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:57:53.751353 4996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:33035
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:53.752765 4996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:53.754706 4996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:53.766937 5004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:53.766960 5012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:53.767009 5002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:53.767899 4996 server_base.cc:1034] running on GCE node
I20260808 03:57:53.768808 4996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:53.770408 4996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:53.771627 4996 hybrid_clock.cc:648] HybridClock initialized: now 1786161473771576 us; error 45 us; skew 500 ppm
I20260808 03:57:53.772001 4996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:57:53.774485 4996 webserver.cc:460] Webserver started at http://127.28.251.254:34999/ using document root <none> and password file <none>
I20260808 03:57:53.775249 4996 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:53.775374 4996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:53.775769 4996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:53.779060 4996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "ae948d608d364a9da118eedfce9ee8d1"
format_stamp: "Formatted at 2026-08-08 03:57:53 on dist-test-slave-pww0"
server_key: "33ae6999b0d8694629b0ec42d682763e"
server_key_iv: "5318982c9e79f5c6029e91e9e5385a29"
server_key_version: "encryptionkey@0"
I20260808 03:57:53.779968 4996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "ae948d608d364a9da118eedfce9ee8d1"
format_stamp: "Formatted at 2026-08-08 03:57:53 on dist-test-slave-pww0"
server_key: "33ae6999b0d8694629b0ec42d682763e"
server_key_iv: "5318982c9e79f5c6029e91e9e5385a29"
server_key_version: "encryptionkey@0"
I20260808 03:57:53.785580 4996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 03:57:53.789414 5021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:53.790858 4996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 03:57:53.791036 4996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "ae948d608d364a9da118eedfce9ee8d1"
format_stamp: "Formatted at 2026-08-08 03:57:53 on dist-test-slave-pww0"
server_key: "33ae6999b0d8694629b0ec42d682763e"
server_key_iv: "5318982c9e79f5c6029e91e9e5385a29"
server_key_version: "encryptionkey@0"
I20260808 03:57:53.791271 4996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:53.797304 4996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:53.798177 4996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:53.798503 4996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:53.817888 4996 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:33035
I20260808 03:57:53.817895 5093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:33035 every 8 connection(s)
I20260808 03:57:53.819434 4996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:53.823678 5094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:53.828379 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 4996
I20260808 03:57:53.828586 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:53.829576 29679 external_mini_cluster.cc:1428] Setting key 198443b39af2436c039ac668fca85c14
I20260808 03:57:53.836961 5094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1: Bootstrap starting.
I20260808 03:57:53.839334 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:53.832420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.28.251.254:33035)
0808 03:57:53.833523 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:53.833538 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:53.833585 (+ 47us) server_negotiation.cc:407] Connection header received
0808 03:57:53.835420 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:53.835447 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:53.835962 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:53.836359 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:53.837544 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:53.837551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:53.837565 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:53.837581 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:53.837646 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:53.837663 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:53.837878 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:53.838020 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 03:57:53.838302 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":393,"threads_started":1}
I20260808 03:57:53.840502 5094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:53.841629 5094 log.cc:824] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:53.844861 5094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1: No bootstrap required, opened a new log
I20260808 03:57:53.852790 5094 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae948d608d364a9da118eedfce9ee8d1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33035 } }
I20260808 03:57:53.853236 5094 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:53.853334 5094 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae948d608d364a9da118eedfce9ee8d1, State: Initialized, Role: FOLLOWER
I20260808 03:57:53.854070 5094 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae948d608d364a9da118eedfce9ee8d1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33035 } }
I20260808 03:57:53.854316 5094 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:53.854497 5094 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:53.854667 5094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:53.857076 5094 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae948d608d364a9da118eedfce9ee8d1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33035 } }
I20260808 03:57:53.857681 5094 leader_election.cc:302] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae948d608d364a9da118eedfce9ee8d1; no voters:
I20260808 03:57:53.858314 5094 leader_election.cc:288] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:53.858467 5108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:53.859226 5108 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [term 1 LEADER]: Becoming Leader. State: Replica: ae948d608d364a9da118eedfce9ee8d1, State: Running, Role: LEADER
I20260808 03:57:53.859860 5108 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae948d608d364a9da118eedfce9ee8d1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33035 } }
I20260808 03:57:53.860627 5094 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:53.862722 5110 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae948d608d364a9da118eedfce9ee8d1. Latest consensus state: current_term: 1 leader_uuid: "ae948d608d364a9da118eedfce9ee8d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae948d608d364a9da118eedfce9ee8d1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33035 } } }
I20260808 03:57:53.862969 5109 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae948d608d364a9da118eedfce9ee8d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae948d608d364a9da118eedfce9ee8d1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33035 } } }
I20260808 03:57:53.863300 5109 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:53.863539 5110 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:53.866987 5116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:53.872601 5116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:53.883183 5116 catalog_manager.cc:1347] Generated new cluster ID: 12bedec2dd8a44bd89851fcbaf126adb
I20260808 03:57:53.883311 5116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:53.898346 5116 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:53.898823 5116 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:53.906348 5116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1: Generated new TSK 0
I20260808 03:57:53.907256 5116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 03:57:53.916383 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:33035
--builtin_ntp_servers=127.28.251.212:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 03:57:54.283720 5139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:54.284036 5139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:54.284165 5139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:54.291888 5139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:54.291999 5139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:54.292145 5139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:57:54.302031 5139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:33035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:54.303507 5139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:54.305369 5139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:54.318847 5147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:54.318830 5149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:54.318933 5145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:54.319936 5139 server_base.cc:1034] running on GCE node
I20260808 03:57:54.320629 5139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:54.321668 5139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:54.322913 5139 hybrid_clock.cc:648] HybridClock initialized: now 1786161474322855 us; error 47 us; skew 500 ppm
I20260808 03:57:54.323294 5139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:57:54.325812 5139 webserver.cc:460] Webserver started at http://127.28.251.193:44583/ using document root <none> and password file <none>
I20260808 03:57:54.326561 5139 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:54.326671 5139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:54.327009 5139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:54.330139 5139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "a4a654f274cc4358936a341470254e14"
format_stamp: "Formatted at 2026-08-08 03:57:54 on dist-test-slave-pww0"
server_key: "69d065eab70213769ea2b4593a216651"
server_key_iv: "9f0781451bd73b3349eab34b16c8f946"
server_key_version: "encryptionkey@0"
I20260808 03:57:54.331017 5139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "a4a654f274cc4358936a341470254e14"
format_stamp: "Formatted at 2026-08-08 03:57:54 on dist-test-slave-pww0"
server_key: "69d065eab70213769ea2b4593a216651"
server_key_iv: "9f0781451bd73b3349eab34b16c8f946"
server_key_version: "encryptionkey@0"
I20260808 03:57:54.336655 5139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 03:57:54.340626 5156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:54.342170 5139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 03:57:54.342370 5139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "a4a654f274cc4358936a341470254e14"
format_stamp: "Formatted at 2026-08-08 03:57:54 on dist-test-slave-pww0"
server_key: "69d065eab70213769ea2b4593a216651"
server_key_iv: "9f0781451bd73b3349eab34b16c8f946"
server_key_version: "encryptionkey@0"
I20260808 03:57:54.342628 5139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:54.348775 5139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:54.349663 5139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:54.349928 5139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:54.350957 5139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:54.352571 5139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:54.352674 5139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:54.352787 5139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:54.352857 5139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:54.385073 5139 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:46781
I20260808 03:57:54.385169 5333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:46781 every 8 connection(s)
I20260808 03:57:54.386443 5139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:57:54.394110 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 5139
I20260808 03:57:54.394306 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:57:54.395143 29679 external_mini_cluster.cc:1428] Setting key 43fa4fc09d28395cb4889e73100b4c7b
I20260808 03:57:54.395429 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:54.389209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:35771 (local address 127.28.251.254:33035)
0808 03:57:54.389693 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:54.389699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:54.391033 (+ 1334us) server_negotiation.cc:407] Connection header received
0808 03:57:54.392835 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:54.392839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:54.392898 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:54.392989 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:54.394581 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:54.394588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:54.394593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:54.394616 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:54.394674 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:54.394691 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:54.394838 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:54.394988 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 03:57:54.395082 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":224,"threads_started":1}
I20260808 03:57:54.396670 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:54.389852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:33035 (local address 127.28.251.193:35771)
0808 03:57:54.390865 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:54.390919 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:54.392547 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:54.393277 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:54.393286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:54.393394 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:54.394201 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:57:54.394204 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:54.394250 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:57:54.394373 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:54.394796 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:54.394806 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:54.394812 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 03:57:54.395041 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 03:57:54.395322 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":282,"threads_started":1}
I20260808 03:57:54.398139 5334 heartbeater.cc:347] Connected to a master server at 127.28.251.254:33035
I20260808 03:57:54.398398 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:33035
--builtin_ntp_servers=127.28.251.212:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 03:57:54.398528 5334 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:54.399289 5334 heartbeater.cc:511] Master 127.28.251.254:33035 requested a full tablet report, sending...
I20260808 03:57:54.401811 5042 ts_manager.cc:194] Registered new tserver with Master: a4a654f274cc4358936a341470254e14 (127.28.251.193:46781)
W20260808 03:57:54.756106 5339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:54.756413 5339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:54.756531 5339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:54.764161 5339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:54.764307 5339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:54.764456 5339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:57:54.774628 5339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:33035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:54.776108 5339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:54.778039 5339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:54.791622 5345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:54.791709 5346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:54.791657 5348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:54.792963 5339 server_base.cc:1034] running on GCE node
I20260808 03:57:54.793648 5339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:54.794694 5339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:54.795933 5339 hybrid_clock.cc:648] HybridClock initialized: now 1786161474795860 us; error 59 us; skew 500 ppm
I20260808 03:57:54.796350 5339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:57:54.798791 5339 webserver.cc:460] Webserver started at http://127.28.251.194:39659/ using document root <none> and password file <none>
I20260808 03:57:54.799599 5339 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:54.799711 5339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:54.800081 5339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:54.803179 5339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "6efc05ff55e64ddba43f0602fe339f44"
format_stamp: "Formatted at 2026-08-08 03:57:54 on dist-test-slave-pww0"
server_key: "724c942c5b5076b1707a2e8fa4266972"
server_key_iv: "2ed89ad35281e6ddf65f38581e0bf3a3"
server_key_version: "encryptionkey@0"
I20260808 03:57:54.804071 5339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "6efc05ff55e64ddba43f0602fe339f44"
format_stamp: "Formatted at 2026-08-08 03:57:54 on dist-test-slave-pww0"
server_key: "724c942c5b5076b1707a2e8fa4266972"
server_key_iv: "2ed89ad35281e6ddf65f38581e0bf3a3"
server_key_version: "encryptionkey@0"
I20260808 03:57:54.809978 5339 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260808 03:57:54.814180 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:54.815791 5339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:57:54.815979 5339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "6efc05ff55e64ddba43f0602fe339f44"
format_stamp: "Formatted at 2026-08-08 03:57:54 on dist-test-slave-pww0"
server_key: "724c942c5b5076b1707a2e8fa4266972"
server_key_iv: "2ed89ad35281e6ddf65f38581e0bf3a3"
server_key_version: "encryptionkey@0"
I20260808 03:57:54.816224 5339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:54.822610 5339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:54.823619 5339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:54.823910 5339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:54.824963 5339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:54.826725 5339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:54.826843 5339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:54.826957 5339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:54.827028 5339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:54.863809 5339 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:37141
I20260808 03:57:54.863866 5530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:37141 every 8 connection(s)
I20260808 03:57:54.865260 5339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:57:54.866463 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 5339
I20260808 03:57:54.866643 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:57:54.867272 29679 external_mini_cluster.cc:1428] Setting key 5866be06717a5c9b5a5004a58e0c4358
I20260808 03:57:54.870822 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:33035
--builtin_ntp_servers=127.28.251.212:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 03:57:54.874738 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:54.868453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:50923 (local address 127.28.251.254:33035)
0808 03:57:54.868632 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:54.868637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:54.870495 (+ 1858us) server_negotiation.cc:407] Connection header received
0808 03:57:54.872406 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:54.872412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:54.872496 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:54.872585 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:54.873965 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:54.873972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:54.873976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:54.873997 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:54.874044 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:54.874057 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:54.874170 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:54.874365 (+ 195us) server_negotiation.cc:299] Negotiation successful
0808 03:57:54.874448 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 03:57:54.875814 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:54.869210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:33035 (local address 127.28.251.194:50923)
0808 03:57:54.870314 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:54.870376 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:54.872111 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:54.872857 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:54.872866 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:54.872970 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:54.873635 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:57:54.873638 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:54.873680 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:57:54.873776 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:54.874148 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:54.874160 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:54.874168 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 03:57:54.874442 (+ 274us) client_negotiation.cc:240] Negotiation successful
0808 03:57:54.874732 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":354,"threads_started":1}
I20260808 03:57:54.877385 5531 heartbeater.cc:347] Connected to a master server at 127.28.251.254:33035
I20260808 03:57:54.877717 5531 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:54.878391 5531 heartbeater.cc:511] Master 127.28.251.254:33035 requested a full tablet report, sending...
I20260808 03:57:54.880034 5042 ts_manager.cc:194] Registered new tserver with Master: 6efc05ff55e64ddba43f0602fe339f44 (127.28.251.194:37141)
W20260808 03:57:55.235989 5534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:55.236327 5534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:55.236474 5534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:55.244215 5534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:55.244356 5534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:55.244513 5534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:57:55.254379 5534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:33035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:55.255868 5534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:55.257822 5534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:55.272053 5546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:55.272094 5544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:55.272140 5542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:55.273312 5534 server_base.cc:1034] running on GCE node
I20260808 03:57:55.273995 5534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:55.275022 5534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:55.276235 5534 hybrid_clock.cc:648] HybridClock initialized: now 1786161475276172 us; error 52 us; skew 500 ppm
I20260808 03:57:55.276587 5534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:57:55.279757 5534 webserver.cc:460] Webserver started at http://127.28.251.195:43845/ using document root <none> and password file <none>
I20260808 03:57:55.280540 5534 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:55.280661 5534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:55.281023 5534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:55.284274 5534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "27c20b5b88974a67a6c63df3e1bb6bf3"
format_stamp: "Formatted at 2026-08-08 03:57:55 on dist-test-slave-pww0"
server_key: "c0504d97ee0a3e15ee94441f918836e5"
server_key_iv: "d4f9eca1f2f98d33da81ac8c08dc2dcb"
server_key_version: "encryptionkey@0"
I20260808 03:57:55.285176 5534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "27c20b5b88974a67a6c63df3e1bb6bf3"
format_stamp: "Formatted at 2026-08-08 03:57:55 on dist-test-slave-pww0"
server_key: "c0504d97ee0a3e15ee94441f918836e5"
server_key_iv: "d4f9eca1f2f98d33da81ac8c08dc2dcb"
server_key_version: "encryptionkey@0"
I20260808 03:57:55.290930 5534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260808 03:57:55.294981 5559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:55.296609 5534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 03:57:55.296794 5534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "27c20b5b88974a67a6c63df3e1bb6bf3"
format_stamp: "Formatted at 2026-08-08 03:57:55 on dist-test-slave-pww0"
server_key: "c0504d97ee0a3e15ee94441f918836e5"
server_key_iv: "d4f9eca1f2f98d33da81ac8c08dc2dcb"
server_key_version: "encryptionkey@0"
I20260808 03:57:55.297039 5534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:55.303181 5534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:55.304126 5534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:55.304409 5534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:55.305403 5534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:57:55.306955 5534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:57:55.307056 5534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:55.307175 5534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:57:55.307245 5534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:55.340346 5534 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:38473
I20260808 03:57:55.340409 5725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:38473 every 8 connection(s)
I20260808 03:57:55.341735 5534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:57:55.349115 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 5534
I20260808 03:57:55.349282 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:57:55.349869 29679 external_mini_cluster.cc:1428] Setting key ea7a67bdc420143fc4be6e35bba21ccf
I20260808 03:57:55.350894 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.344472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:60049 (local address 127.28.251.254:33035)
0808 03:57:55.344707 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:55.344712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:55.346325 (+ 1613us) server_negotiation.cc:407] Connection header received
0808 03:57:55.348367 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:55.348374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:55.348462 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:55.348565 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:55.350120 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:55.350125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:55.350129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:55.350146 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:55.350180 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:55.350192 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:55.350304 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:55.350516 (+ 212us) server_negotiation.cc:299] Negotiation successful
0808 03:57:55.350592 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 03:57:55.352159 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.345025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:33035 (local address 127.28.251.195:60049)
0808 03:57:55.346140 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:55.346188 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:55.348056 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:55.348883 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:55.348891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:55.349005 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:55.349786 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:57:55.349789 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:55.349831 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:57:55.349934 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:55.350283 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:55.350294 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:55.350302 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 03:57:55.350589 (+ 287us) client_negotiation.cc:240] Negotiation successful
0808 03:57:55.350897 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":331,"threads_started":1}
I20260808 03:57:55.353442 5727 heartbeater.cc:347] Connected to a master server at 127.28.251.254:33035
I20260808 03:57:55.353752 5727 heartbeater.cc:464] Registering TS with master...
I20260808 03:57:55.354367 5727 heartbeater.cc:511] Master 127.28.251.254:33035 requested a full tablet report, sending...
I20260808 03:57:55.355922 5042 ts_manager.cc:194] Registered new tserver with Master: 27c20b5b88974a67a6c63df3e1bb6bf3 (127.28.251.195:38473)
I20260808 03:57:55.365891 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 03:57:55.373335 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.370670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.28.251.254:33035)
0808 03:57:55.370859 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:55.370864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:55.371312 (+ 448us) server_negotiation.cc:407] Connection header received
0808 03:57:55.371491 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:55.371497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:55.371567 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:55.371657 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:55.372753 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:55.372758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:55.372761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:55.372778 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:55.372813 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:55.372826 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:55.372927 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:55.373043 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 03:57:55.373095 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 03:57:55.377815 5042 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:57:55.385177 5042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:57:55.405328 5334 heartbeater.cc:503] Master 127.28.251.254:33035 was elected leader, sending a full tablet report...
I20260808 03:57:55.406917 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.398433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:38473 (local address 127.0.0.1:35602)
0808 03:57:55.401228 (+ 2795us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:55.401248 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:55.401364 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:55.403411 (+ 2047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:55.403421 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:55.403566 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:55.404517 (+ 951us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:57:55.404522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:55.404559 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:57:55.404660 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:55.406079 (+ 1419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:55.406095 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:55.406105 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 03:57:55.406324 (+ 219us) client_negotiation.cc:240] Negotiation successful
0808 03:57:55.406436 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2629,"thread_start_us":2498,"threads_started":1}
I20260808 03:57:55.407732 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.399119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.28.251.195:38473)
0808 03:57:55.400677 (+ 1558us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:55.400687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:55.401494 (+ 807us) server_negotiation.cc:407] Connection header received
0808 03:57:55.401583 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:55.401592 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:55.401861 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:55.402074 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:55.404892 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:55.404904 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:55.404924 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:55.404947 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:55.405769 (+ 822us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:55.405808 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:55.406013 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:55.407160 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0808 03:57:55.407297 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1360,"thread_start_us":1166,"threads_started":1}
I20260808 03:57:55.413014 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.398106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46781 (local address 127.0.0.1:46124)
0808 03:57:55.399146 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:55.399204 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:55.399353 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:55.409614 (+ 10261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:55.409620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:55.409650 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:55.410537 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:57:55.410541 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:55.410562 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:57:55.410621 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:55.412288 (+ 1667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:55.412294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:55.412300 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 03:57:55.412444 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 03:57:55.412513 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":559,"threads_started":1}
I20260808 03:57:55.413973 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.398300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.28.251.193:46781)
0808 03:57:55.409007 (+ 10707us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:55.409016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:55.409047 (+ 31us) server_negotiation.cc:407] Connection header received
0808 03:57:55.409115 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:55.409125 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:55.409408 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:55.410829 (+ 1421us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:55.412005 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:55.412014 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:55.412029 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:55.412047 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:55.412110 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:55.412131 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:55.413419 (+ 1288us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:55.413506 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 03:57:55.413620 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10529,"thread_start_us":10379,"threads_started":1}
I20260808 03:57:55.413168 5626 tablet_service.cc:1467] Processing CreateTablet for tablet 39e8b9e077b646a09b86f3b0a46b639a (DEFAULT_TABLE table=test-table [id=e46d01f13fd14e1c99b3c5ce6e0c1990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:55.415355 5626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e8b9e077b646a09b86f3b0a46b639a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:55.416260 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.400598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:37141 (local address 127.0.0.1:58846)
0808 03:57:55.408108 (+ 7510us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:55.408129 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:55.408233 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:55.413926 (+ 5693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:55.413931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:55.413978 (+ 47us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:55.414568 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:57:55.414571 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:55.414589 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:57:55.414645 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:55.415892 (+ 1247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:55.415897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:55.415901 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 03:57:55.415977 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 03:57:55.416031 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7396,"thread_start_us":14084,"threads_started":1}
I20260808 03:57:55.416887 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.400756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.28.251.194:37141)
0808 03:57:55.413105 (+ 12349us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:55.413114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:55.413146 (+ 32us) server_negotiation.cc:407] Connection header received
0808 03:57:55.413217 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:55.413226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:55.413488 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:55.414363 (+ 875us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:55.415625 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:55.415633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:55.415647 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:55.415691 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:55.415745 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:55.415765 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:55.416496 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:55.416565 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 03:57:55.416642 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12152,"thread_start_us":12852,"threads_started":1}
I20260808 03:57:55.420203 5230 tablet_service.cc:1467] Processing CreateTablet for tablet 39e8b9e077b646a09b86f3b0a46b639a (DEFAULT_TABLE table=test-table [id=e46d01f13fd14e1c99b3c5ce6e0c1990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:55.420486 5432 tablet_service.cc:1467] Processing CreateTablet for tablet 39e8b9e077b646a09b86f3b0a46b639a (DEFAULT_TABLE table=test-table [id=e46d01f13fd14e1c99b3c5ce6e0c1990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:57:55.421710 5432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e8b9e077b646a09b86f3b0a46b639a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:55.421795 5230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e8b9e077b646a09b86f3b0a46b639a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:55.432287 5753 tablet_bootstrap.cc:492] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: Bootstrap starting.
I20260808 03:57:55.434412 5756 tablet_bootstrap.cc:492] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: Bootstrap starting.
I20260808 03:57:55.437129 5761 tablet_bootstrap.cc:492] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: Bootstrap starting.
I20260808 03:57:55.437379 5753 tablet_bootstrap.cc:654] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:55.438316 5756 tablet_bootstrap.cc:654] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:55.438822 5753 log.cc:824] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:55.439316 5756 log.cc:824] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:55.440656 5761 tablet_bootstrap.cc:654] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:55.441882 5761 log.cc:824] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:55.442876 5753 tablet_bootstrap.cc:492] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: No bootstrap required, opened a new log
I20260808 03:57:55.443270 5753 ts_tablet_manager.cc:1397] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260808 03:57:55.448618 5756 tablet_bootstrap.cc:492] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: No bootstrap required, opened a new log
I20260808 03:57:55.448887 5761 tablet_bootstrap.cc:492] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: No bootstrap required, opened a new log
I20260808 03:57:55.448923 5756 ts_tablet_manager.cc:1397] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260808 03:57:55.449196 5761 ts_tablet_manager.cc:1397] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260808 03:57:55.454408 5753 raft_consensus.cc:348] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a654f274cc4358936a341470254e14" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46781 } } peers { permanent_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38473 } } peers { permanent_uuid: "6efc05ff55e64ddba43f0602fe339f44" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37141 } }
I20260808 03:57:55.454943 5753 raft_consensus.cc:374] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:55.455101 5753 raft_consensus.cc:729] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c20b5b88974a67a6c63df3e1bb6bf3, State: Initialized, Role: FOLLOWER
I20260808 03:57:55.455823 5753 consensus_queue.cc:260] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a654f274cc4358936a341470254e14" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46781 } } peers { permanent_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38473 } } peers { permanent_uuid: "6efc05ff55e64ddba43f0602fe339f44" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37141 } }
I20260808 03:57:55.457109 5727 heartbeater.cc:503] Master 127.28.251.254:33035 was elected leader, sending a full tablet report...
I20260808 03:57:55.457217 5756 raft_consensus.cc:348] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a654f274cc4358936a341470254e14" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46781 } } peers { permanent_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38473 } } peers { permanent_uuid: "6efc05ff55e64ddba43f0602fe339f44" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37141 } }
I20260808 03:57:55.457664 5756 raft_consensus.cc:374] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:55.457754 5753 ts_tablet_manager.cc:1428] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20260808 03:57:55.457758 5756 raft_consensus.cc:729] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efc05ff55e64ddba43f0602fe339f44, State: Initialized, Role: FOLLOWER
I20260808 03:57:55.458388 5756 consensus_queue.cc:260] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a654f274cc4358936a341470254e14" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46781 } } peers { permanent_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38473 } } peers { permanent_uuid: "6efc05ff55e64ddba43f0602fe339f44" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37141 } }
I20260808 03:57:55.459738 5531 heartbeater.cc:503] Master 127.28.251.254:33035 was elected leader, sending a full tablet report...
I20260808 03:57:55.459316 5761 raft_consensus.cc:348] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a654f274cc4358936a341470254e14" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46781 } } peers { permanent_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38473 } } peers { permanent_uuid: "6efc05ff55e64ddba43f0602fe339f44" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37141 } }
I20260808 03:57:55.459919 5761 raft_consensus.cc:374] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:55.460026 5761 raft_consensus.cc:729] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a654f274cc4358936a341470254e14, State: Initialized, Role: FOLLOWER
I20260808 03:57:55.460724 5756 ts_tablet_manager.cc:1428] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
I20260808 03:57:55.460744 5761 consensus_queue.cc:260] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a654f274cc4358936a341470254e14" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46781 } } peers { permanent_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38473 } } peers { permanent_uuid: "6efc05ff55e64ddba43f0602fe339f44" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37141 } }
I20260808 03:57:55.462697 5761 ts_tablet_manager.cc:1428] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260808 03:57:55.593493 5728 tablet.cc:2369] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:55.603708 5769 raft_consensus.cc:482] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:57:55.604082 5769 raft_consensus.cc:504] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a654f274cc4358936a341470254e14" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46781 } } peers { permanent_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38473 } } peers { permanent_uuid: "6efc05ff55e64ddba43f0602fe339f44" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37141 } }
I20260808 03:57:55.606047 5769 leader_election.cc:288] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4a654f274cc4358936a341470254e14 (127.28.251.193:46781), 6efc05ff55e64ddba43f0602fe339f44 (127.28.251.194:37141)
I20260808 03:57:55.608645 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.606550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:60273 (local address 127.28.251.193:46781)
0808 03:57:55.606747 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:55.606751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:55.606767 (+ 16us) server_negotiation.cc:407] Connection header received
0808 03:57:55.606828 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:55.606833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:55.606895 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:55.606996 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:55.608044 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:55.608049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:55.608052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:55.608068 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:55.608100 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:55.608112 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:55.608225 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:55.608321 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 03:57:55.608384 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 03:57:55.608652 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.606324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46781 (local address 127.28.251.195:60273)
0808 03:57:55.606507 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:55.606523 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:55.606612 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:55.607194 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:55.607199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:55.607222 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:55.607833 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:57:55.607835 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:55.607851 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:57:55.607894 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:55.608203 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:55.608208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:55.608213 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 03:57:55.608315 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 03:57:55.608381 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 03:57:55.609409 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.606953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:51655 (local address 127.28.251.194:37141)
0808 03:57:55.607144 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:55.607149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:55.607548 (+ 399us) server_negotiation.cc:407] Connection header received
0808 03:57:55.607703 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:55.607708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:55.607760 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:55.607843 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:55.608774 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:55.608781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:55.608786 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:55.608810 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:55.608852 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:55.608866 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:55.608978 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:55.609048 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 03:57:55.609109 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 03:57:55.609509 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.606771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:37141 (local address 127.28.251.195:51655)
0808 03:57:55.607423 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:55.607439 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:55.607576 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:55.607900 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:55.607904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:55.607928 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:57:55.608554 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:57:55.608557 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:57:55.608571 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:57:55.608609 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:57:55.608950 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:57:55.608955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:57:55.608959 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 03:57:55.609066 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 03:57:55.609115 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":327,"threads_started":1}
I20260808 03:57:55.609795 5261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e8b9e077b646a09b86f3b0a46b639a" candidate_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a654f274cc4358936a341470254e14" is_pre_election: true
I20260808 03:57:55.610412 5261 raft_consensus.cc:2454] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27c20b5b88974a67a6c63df3e1bb6bf3 in term 0.
I20260808 03:57:55.610555 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e8b9e077b646a09b86f3b0a46b639a" candidate_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efc05ff55e64ddba43f0602fe339f44" is_pre_election: true
I20260808 03:57:55.611065 5464 raft_consensus.cc:2454] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27c20b5b88974a67a6c63df3e1bb6bf3 in term 0.
I20260808 03:57:55.611079 5567 leader_election.cc:302] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27c20b5b88974a67a6c63df3e1bb6bf3, a4a654f274cc4358936a341470254e14; no voters:
I20260808 03:57:55.611441 5769 raft_consensus.cc:2781] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:57:55.611639 5769 raft_consensus.cc:482] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:57:55.611750 5769 raft_consensus.cc:3037] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:55.614018 5769 raft_consensus.cc:504] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a654f274cc4358936a341470254e14" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46781 } } peers { permanent_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38473 } } peers { permanent_uuid: "6efc05ff55e64ddba43f0602fe339f44" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37141 } }
I20260808 03:57:55.614614 5769 leader_election.cc:288] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [CANDIDATE]: Term 1 election: Requested vote from peers a4a654f274cc4358936a341470254e14 (127.28.251.193:46781), 6efc05ff55e64ddba43f0602fe339f44 (127.28.251.194:37141)
I20260808 03:57:55.615183 5261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e8b9e077b646a09b86f3b0a46b639a" candidate_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a654f274cc4358936a341470254e14"
I20260808 03:57:55.615239 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e8b9e077b646a09b86f3b0a46b639a" candidate_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efc05ff55e64ddba43f0602fe339f44"
I20260808 03:57:55.615392 5261 raft_consensus.cc:3037] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:55.615432 5464 raft_consensus.cc:3037] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44 [term 0 FOLLOWER]: Advancing to term 1
W20260808 03:57:55.617554 5532 tablet.cc:2369] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:55.617743 5261 raft_consensus.cc:2454] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27c20b5b88974a67a6c63df3e1bb6bf3 in term 1.
I20260808 03:57:55.618078 5464 raft_consensus.cc:2454] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27c20b5b88974a67a6c63df3e1bb6bf3 in term 1.
I20260808 03:57:55.618204 5567 leader_election.cc:302] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27c20b5b88974a67a6c63df3e1bb6bf3, a4a654f274cc4358936a341470254e14; no voters:
I20260808 03:57:55.618551 5769 raft_consensus.cc:2781] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:55.619004 5769 raft_consensus.cc:686] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 1 LEADER]: Becoming Leader. State: Replica: 27c20b5b88974a67a6c63df3e1bb6bf3, State: Running, Role: LEADER
I20260808 03:57:55.619515 5769 consensus_queue.cc:237] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a654f274cc4358936a341470254e14" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46781 } } peers { permanent_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38473 } } peers { permanent_uuid: "6efc05ff55e64ddba43f0602fe339f44" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37141 } }
I20260808 03:57:55.624133 5038 catalog_manager.cc:5495] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27c20b5b88974a67a6c63df3e1bb6bf3 (127.28.251.195). New cstate: current_term: 1 leader_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a654f274cc4358936a341470254e14" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27c20b5b88974a67a6c63df3e1bb6bf3" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6efc05ff55e64ddba43f0602fe339f44" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37141 } health_report { overall_health: UNKNOWN } } }
W20260808 03:57:55.639066 5335 tablet.cc:2369] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:57:55.661670 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.659314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.28.251.195:38473)
0808 03:57:55.659589 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:55.659595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:55.659619 (+ 24us) server_negotiation.cc:407] Connection header received
0808 03:57:55.659907 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:55.659912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:55.660007 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:55.660103 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:55.661080 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:55.661085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:55.661089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:55.661108 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:55.661148 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:55.661161 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:55.661283 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:55.661341 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 03:57:55.661395 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260808 03:57:55.670984 5261 raft_consensus.cc:1261] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14 [term 1 FOLLOWER]: Refusing update from remote peer 27c20b5b88974a67a6c63df3e1bb6bf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:55.670982 5464 raft_consensus.cc:1261] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44 [term 1 FOLLOWER]: Refusing update from remote peer 27c20b5b88974a67a6c63df3e1bb6bf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:57:55.672099 5777 consensus_queue.cc:1035] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a654f274cc4358936a341470254e14" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:55.672544 5769 consensus_queue.cc:1035] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efc05ff55e64ddba43f0602fe339f44" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:57:55.683843 5783 mvcc.cc:205] Tried to move back new op lower bound from 7316117404337192960 to 7316117404141342720. Current Snapshot: MvccSnapshot[applied={T|T < 7316117404337192960}]
I20260808 03:57:55.697901 5039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:55.698294 5039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:57:55.702674 5039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae948d608d364a9da118eedfce9ee8d1: Sending DeleteTablet for 3 replicas of tablet 39e8b9e077b646a09b86f3b0a46b639a
I20260808 03:57:55.704016 5230 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e8b9e077b646a09b86f3b0a46b639a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:55 UTC) from {username='slave'} at 127.0.0.1:46124
I20260808 03:57:55.704130 5626 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e8b9e077b646a09b86f3b0a46b639a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:55 UTC) from {username='slave'} at 127.0.0.1:35602
I20260808 03:57:55.704936 5799 tablet_replica.cc:330] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: stopping tablet replica
I20260808 03:57:55.705484 5799 raft_consensus.cc:2229] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:55.705570 5432 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e8b9e077b646a09b86f3b0a46b639a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:57:55 UTC) from {username='slave'} at 127.0.0.1:58846
I20260808 03:57:55.706175 5799 raft_consensus.cc:2258] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:55.708881 5799 ts_tablet_manager.cc:1905] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:55.709702 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 5139
I20260808 03:57:55.716064 5800 tablet_replica.cc:330] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: stopping tablet replica
I20260808 03:57:55.716629 5800 raft_consensus.cc:2229] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:57:55.717377 5800 raft_consensus.cc:2258] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:55.718241 5801 tablet_replica.cc:330] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: stopping tablet replica
I20260808 03:57:55.718812 5801 raft_consensus.cc:2229] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:57:55.719494 5801 raft_consensus.cc:2258] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:57:55.720001 5800 ts_tablet_manager.cc:1905] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:55.723299 5801 ts_tablet_manager.cc:1905] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:57:55.724829 5799 ts_tablet_manager.cc:1918] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:57:55.728623 5800 ts_tablet_manager.cc:1918] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:57:55.728866 5800 log.cc:1196] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/39e8b9e077b646a09b86f3b0a46b639a
I20260808 03:57:55.729539 5800 ts_tablet_manager.cc:1939] T 39e8b9e077b646a09b86f3b0a46b639a P 27c20b5b88974a67a6c63df3e1bb6bf3: Deleting consensus metadata
I20260808 03:57:55.731223 5801 ts_tablet_manager.cc:1918] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:57:55.731434 5801 log.cc:1196] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/39e8b9e077b646a09b86f3b0a46b639a
I20260808 03:57:55.731716 5799 log.cc:1196] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/39e8b9e077b646a09b86f3b0a46b639a
I20260808 03:57:55.731968 5022 catalog_manager.cc:4841] TS 27c20b5b88974a67a6c63df3e1bb6bf3 (127.28.251.195:38473): tablet 39e8b9e077b646a09b86f3b0a46b639a (table test-table [id=e46d01f13fd14e1c99b3c5ce6e0c1990]) successfully deleted
I20260808 03:57:55.727636 5747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.711274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.28.251.193:46781)
0808 03:57:55.711602 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:55.711608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:55.724679 (+ 13071us) server_negotiation.cc:407] Connection header received
0808 03:57:55.725247 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:55.725253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:55.725370 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:55.725455 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:55.726750 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:55.726757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:55.726761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:55.726786 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:55.726843 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:55.726860 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:55.726965 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:55.727215 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 03:57:55.727292 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 03:57:55.732347 5799 ts_tablet_manager.cc:1939] T 39e8b9e077b646a09b86f3b0a46b639a P a4a654f274cc4358936a341470254e14: Deleting consensus metadata
I20260808 03:57:55.733234 5324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:55.734021 5801 ts_tablet_manager.cc:1939] T 39e8b9e077b646a09b86f3b0a46b639a P 6efc05ff55e64ddba43f0602fe339f44: Deleting consensus metadata
I20260808 03:57:55.736078 5025 catalog_manager.cc:4841] TS 6efc05ff55e64ddba43f0602fe339f44 (127.28.251.194:37141): tablet 39e8b9e077b646a09b86f3b0a46b639a (table test-table [id=e46d01f13fd14e1c99b3c5ce6e0c1990]) successfully deleted
I20260808 03:57:55.824512 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 5139
I20260808 03:57:55.824488 5025 catalog_manager.cc:4841] TS a4a654f274cc4358936a341470254e14 (127.28.251.193:46781): tablet 39e8b9e077b646a09b86f3b0a46b639a (table test-table [id=e46d01f13fd14e1c99b3c5ce6e0c1990]) successfully deleted
I20260808 03:57:55.854499 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 5339
I20260808 03:57:55.857589 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:55.855446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.28.251.194:37141)
0808 03:57:55.855644 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:55.855649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:55.855671 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:57:55.855807 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:55.855812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:55.855914 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:55.856001 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:55.856975 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:55.856981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:55.856984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:55.857000 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:55.857033 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:55.857045 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:55.857144 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:55.857239 (+ 95us) server_negotiation.cc:299] Negotiation successful
0808 03:57:55.857290 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 03:57:55.858327 5524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:55.976411 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 5339
I20260808 03:57:56.011121 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 5534
I20260808 03:57:56.014417 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:56.012265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.28.251.195:38473)
0808 03:57:56.012495 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:56.012501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:56.012522 (+ 21us) server_negotiation.cc:407] Connection header received
0808 03:57:56.012612 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:56.012616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:56.012676 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:56.012767 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:56.013742 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:56.013747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:56.013750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:56.013767 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:56.013803 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:56.013816 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:56.013957 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:56.014040 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 03:57:56.014105 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 03:57:56.015230 5720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:56.162976 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 5534
I20260808 03:57:56.196283 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 4996
I20260808 03:57:56.197535 5087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:56.310911 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 4996
2026-08-08T03:57:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3018 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 03:57:56.363227 29679 mini_oidc.cc:152] Starting JWKS server
I20260808 03:57:56.366379 29679 webserver.cc:460] Webserver started at https://127.0.0.1:39249/ using document root <none> and password file <none>
I20260808 03:57:56.366767 29679 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 03:57:56.367446 29679 webserver.cc:460] Webserver started at http://127.0.0.1:43101/ using document root <none> and password file <none>
2026-08-08T03:57:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:56Z Disabled control of system clock
I20260808 03:57:56.407043 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41003
--webserver_interface=127.28.251.254
--webserver_port=0
--builtin_ntp_servers=127.28.251.212:38169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:41003
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39249/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/testchainca.cert with env {}
W20260808 03:57:56.765514 5819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:56.765895 5819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:56.766034 5819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:56.773765 5819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:56.773926 5819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:56.773991 5819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:56.774077 5819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:56.774139 5819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 03:57:56.774232 5819 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 03:57:56.774291 5819 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39249/jwks.json
I20260808 03:57:56.784922 5819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:38169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:41003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39249/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:56.786419 5819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:56.787878 5819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:56.800386 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:56.800323 5833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:56.800356 5835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:56.801255 5819 server_base.cc:1034] running on GCE node
I20260808 03:57:56.802122 5819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:56.803690 5819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:56.804949 5819 hybrid_clock.cc:648] HybridClock initialized: now 1786161476804905 us; error 58 us; skew 500 ppm
I20260808 03:57:56.805337 5819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:57:56.810627 5819 webserver.cc:460] Webserver started at http://127.28.251.254:34097/ using document root <none> and password file <none>
I20260808 03:57:56.811429 5819 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:56.811609 5819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:56.811949 5819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:56.815191 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "ebc008f0c9c14d24b7feb0eabea778d7"
format_stamp: "Formatted at 2026-08-08 03:57:56 on dist-test-slave-pww0"
server_key: "0a0522c6e2dc72c6e8f01f27a8ce1b64"
server_key_iv: "540f8aab758d6db4b2308511539f7e39"
server_key_version: "encryptionkey@0"
I20260808 03:57:56.816107 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "ebc008f0c9c14d24b7feb0eabea778d7"
format_stamp: "Formatted at 2026-08-08 03:57:56 on dist-test-slave-pww0"
server_key: "0a0522c6e2dc72c6e8f01f27a8ce1b64"
server_key_iv: "540f8aab758d6db4b2308511539f7e39"
server_key_version: "encryptionkey@0"
I20260808 03:57:56.821630 5819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 03:57:56.825383 5841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:56.826869 5819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 03:57:56.827071 5819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "ebc008f0c9c14d24b7feb0eabea778d7"
format_stamp: "Formatted at 2026-08-08 03:57:56 on dist-test-slave-pww0"
server_key: "0a0522c6e2dc72c6e8f01f27a8ce1b64"
server_key_iv: "540f8aab758d6db4b2308511539f7e39"
server_key_version: "encryptionkey@0"
I20260808 03:57:56.827315 5819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:56.843930 5819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:56.844862 5819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:56.845177 5819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:56.864200 5819 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:41003
I20260808 03:57:56.864236 5929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:41003 every 8 connection(s)
I20260808 03:57:56.865664 5819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:56.869843 5930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:56.874200 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 5819
I20260808 03:57:56.874464 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:56.875291 29679 external_mini_cluster.cc:1428] Setting key 202f08ecc8f658ecc2da350d82e4314e
I20260808 03:57:56.882922 5930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7: Bootstrap starting.
I20260808 03:57:56.886363 5930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:56.887651 5930 log.cc:824] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:56.890864 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:56.877684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.28.251.254:41003)
0808 03:57:56.878867 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:56.878879 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:56.878929 (+ 50us) server_negotiation.cc:407] Connection header received
0808 03:57:56.881381 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:56.881413 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:56.881982 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:56.882402 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:56.883028 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:56.884435 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:56.885857 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:56.886654 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:56.886888 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:56.888169 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:56.888180 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:56.888201 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:56.889022 (+ 821us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:56.889140 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:56.889169 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:56.889286 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:56.889531 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 03:57:56.889879 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":407,"threads_started":1}
I20260808 03:57:56.890920 5930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7: No bootstrap required, opened a new log
I20260808 03:57:56.894984 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 03:57:56.899561 5930 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc008f0c9c14d24b7feb0eabea778d7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } }
I20260808 03:57:56.899989 5930 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:56.900096 5930 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc008f0c9c14d24b7feb0eabea778d7, State: Initialized, Role: FOLLOWER
I20260808 03:57:56.900735 5930 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc008f0c9c14d24b7feb0eabea778d7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } }
I20260808 03:57:56.900979 5930 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:56.901155 5930 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:56.901384 5930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:56.903748 5930 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc008f0c9c14d24b7feb0eabea778d7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } }
I20260808 03:57:56.904318 5930 leader_election.cc:302] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebc008f0c9c14d24b7feb0eabea778d7; no voters:
I20260808 03:57:56.904938 5930 leader_election.cc:288] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:56.905092 5940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:56.905876 5940 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [term 1 LEADER]: Becoming Leader. State: Replica: ebc008f0c9c14d24b7feb0eabea778d7, State: Running, Role: LEADER
I20260808 03:57:56.906432 5940 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc008f0c9c14d24b7feb0eabea778d7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } }
I20260808 03:57:56.907171 5930 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:56.909639 5942 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebc008f0c9c14d24b7feb0eabea778d7. Latest consensus state: current_term: 1 leader_uuid: "ebc008f0c9c14d24b7feb0eabea778d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc008f0c9c14d24b7feb0eabea778d7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } } }
I20260808 03:57:56.910109 5942 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:56.911042 5951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:56.910844 5941 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebc008f0c9c14d24b7feb0eabea778d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc008f0c9c14d24b7feb0eabea778d7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } } }
I20260808 03:57:56.911626 5941 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:56.916735 5951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:56.926060 5951 catalog_manager.cc:1347] Generated new cluster ID: f15517a6ecaa44bda4ee58e99dd7e85a
I20260808 03:57:56.926193 5951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:56.946760 5951 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:56.948799 5951 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:56.960177 5951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebc008f0c9c14d24b7feb0eabea778d7: Generated new TSK 0
I20260808 03:57:56.961409 5951 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 03:57:57.016145 5976 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 03:57:57.016395 5976 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 03:57:57.018358 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:56.987022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.28.251.254:41003)
0808 03:57:56.987259 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:56.987267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:56.987693 (+ 426us) server_negotiation.cc:407] Connection header received
0808 03:57:56.988140 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:56.988146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:56.988238 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:56.988387 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0808 03:57:56.988983 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:56.989748 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:56.991279 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:56.991803 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:56.991942 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:56.992544 (+ 602us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 03:57:57.017599 (+ 25055us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 03:57:57.018025 (+ 426us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 03:57:57.018515 5934 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 03:57:57.020283 5975 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:57:56.986874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41003 (local address 127.0.0.1:48614)
0808 03:57:56.987576 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:56.987593 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:56.987796 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:56.988436 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:56.988442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:56.988455 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0808 03:57:56.988808 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:56.988826 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:56.989913 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:56.989917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:56.991111 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:56.991125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:56.992004 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:56.992012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:56.992158 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:56.992228 (+ 70us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 03:57:57.017961 (+ 25733us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 03:57:57.018380 (+ 419us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.251.254:41003: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":422,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.251.254:41003: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 03:57:57.022257 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 5819
I20260808 03:57:57.023548 5923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:57.122673 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 5819
2026-08-08T03:57:57Z chronyd exiting
I20260808 03:57:57.180127 29679 test_util.cc:175] -----------------------------------------------
I20260808 03:57:57.180297 29679 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786161430375636-29679-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (822 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 03:57:57.184609 29679 mini_oidc.cc:152] Starting JWKS server
I20260808 03:57:57.186544 29679 webserver.cc:460] Webserver started at https://127.0.0.1:40571/ using document root <none> and password file <none>
I20260808 03:57:57.186894 29679 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 03:57:57.187772 29679 webserver.cc:460] Webserver started at http://127.0.0.1:33169/ using document root <none> and password file <none>
2026-08-08T03:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:57Z Disabled control of system clock
I20260808 03:57:57.227854 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43295
--webserver_interface=127.28.251.254
--webserver_port=0
--builtin_ntp_servers=127.28.251.212:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:43295
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40571/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/testchainca.cert with env {}
W20260808 03:57:57.578982 5987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:57.579329 5987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:57.579443 5987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:57.586975 5987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:57.587100 5987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:57.587174 5987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:57.587255 5987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:57.587324 5987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 03:57:57.587414 5987 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 03:57:57.587491 5987 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40571/jwks.json
I20260808 03:57:57.598033 5987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:43295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40571/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:57.599519 5987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:57.600911 5987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:57.613312 5999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:57.613276 5997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:57.613205 5994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:57.614480 5987 server_base.cc:1034] running on GCE node
I20260808 03:57:57.615378 5987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:57.617034 5987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:57.618270 5987 hybrid_clock.cc:648] HybridClock initialized: now 1786161477618220 us; error 39 us; skew 500 ppm
I20260808 03:57:57.618656 5987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:57:57.621124 5987 webserver.cc:460] Webserver started at http://127.28.251.254:34737/ using document root <none> and password file <none>
I20260808 03:57:57.621903 5987 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:57.622123 5987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:57.622498 5987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:57.625677 5987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "befab98177de4df7a3ec8a835e9c9b38"
format_stamp: "Formatted at 2026-08-08 03:57:57 on dist-test-slave-pww0"
server_key: "371b04843a48ed123f7396a09f2fa16b"
server_key_iv: "1a95dea698fd7d0185206a5f0ea83c4e"
server_key_version: "encryptionkey@0"
I20260808 03:57:57.626523 5987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "befab98177de4df7a3ec8a835e9c9b38"
format_stamp: "Formatted at 2026-08-08 03:57:57 on dist-test-slave-pww0"
server_key: "371b04843a48ed123f7396a09f2fa16b"
server_key_iv: "1a95dea698fd7d0185206a5f0ea83c4e"
server_key_version: "encryptionkey@0"
I20260808 03:57:57.632087 5987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260808 03:57:57.635906 6010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:57.637405 5987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 03:57:57.637609 5987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "befab98177de4df7a3ec8a835e9c9b38"
format_stamp: "Formatted at 2026-08-08 03:57:57 on dist-test-slave-pww0"
server_key: "371b04843a48ed123f7396a09f2fa16b"
server_key_iv: "1a95dea698fd7d0185206a5f0ea83c4e"
server_key_version: "encryptionkey@0"
I20260808 03:57:57.637848 5987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:57.657450 5987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:57.658442 5987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:57.658776 5987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:57.677521 5987 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:43295
I20260808 03:57:57.677528 6082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:43295 every 8 connection(s)
I20260808 03:57:57.679073 5987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:57.683355 6085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:57.684551 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 5987
I20260808 03:57:57.684726 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:57.685413 29679 external_mini_cluster.cc:1428] Setting key 1d312eae1062c7381559bc8ab5058b41
I20260808 03:57:57.697890 6085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38: Bootstrap starting.
I20260808 03:57:57.699985 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:57.687395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.28.251.254:43295)
0808 03:57:57.688550 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:57.688562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:57.688608 (+ 46us) server_negotiation.cc:407] Connection header received
0808 03:57:57.690253 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:57.690279 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:57.690826 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:57.691182 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:57.691843 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:57.693138 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:57.694994 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:57.695827 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:57.695973 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:57.697304 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:57.697313 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:57.697329 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:57.697920 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:57.698031 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:57.698064 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:57.698200 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:57.698474 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 03:57:57.698841 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":392,"threads_started":1}
I20260808 03:57:57.701476 6085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:57.702638 6085 log.cc:824] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:57.704136 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 03:57:57.705819 6085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38: No bootstrap required, opened a new log
I20260808 03:57:57.714229 6085 raft_consensus.cc:348] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befab98177de4df7a3ec8a835e9c9b38" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43295 } }
I20260808 03:57:57.714744 6085 raft_consensus.cc:374] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:57.714879 6085 raft_consensus.cc:729] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: befab98177de4df7a3ec8a835e9c9b38, State: Initialized, Role: FOLLOWER
I20260808 03:57:57.715600 6085 consensus_queue.cc:260] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befab98177de4df7a3ec8a835e9c9b38" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43295 } }
I20260808 03:57:57.715850 6085 raft_consensus.cc:388] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:57.715991 6085 raft_consensus.cc:482] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:57.716156 6085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:57.718416 6085 raft_consensus.cc:504] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befab98177de4df7a3ec8a835e9c9b38" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43295 } }
I20260808 03:57:57.718978 6085 leader_election.cc:302] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: befab98177de4df7a3ec8a835e9c9b38; no voters:
I20260808 03:57:57.719604 6085 leader_election.cc:288] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:57.719762 6093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:57.720805 6093 raft_consensus.cc:686] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [term 1 LEADER]: Becoming Leader. State: Replica: befab98177de4df7a3ec8a835e9c9b38, State: Running, Role: LEADER
I20260808 03:57:57.721436 6093 consensus_queue.cc:237] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befab98177de4df7a3ec8a835e9c9b38" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43295 } }
I20260808 03:57:57.721917 6085 sys_catalog.cc:564] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:57.724395 6096 sys_catalog.cc:455] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader befab98177de4df7a3ec8a835e9c9b38. Latest consensus state: current_term: 1 leader_uuid: "befab98177de4df7a3ec8a835e9c9b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befab98177de4df7a3ec8a835e9c9b38" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43295 } } }
I20260808 03:57:57.725435 6096 sys_catalog.cc:458] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:57.724373 6094 sys_catalog.cc:455] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "befab98177de4df7a3ec8a835e9c9b38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "befab98177de4df7a3ec8a835e9c9b38" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43295 } } }
I20260808 03:57:57.726049 6094 sys_catalog.cc:458] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:57.727046 6107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:57.733394 6107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:57.742442 6107 catalog_manager.cc:1347] Generated new cluster ID: 40ca724579764ac7b37905d01d95fb15
I20260808 03:57:57.742614 6107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:57.764567 6107 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:57.766544 6107 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:57.779834 6107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P befab98177de4df7a3ec8a835e9c9b38: Generated new TSK 0
I20260808 03:57:57.781078 6107 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 03:57:57.816147 6132 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 03:57:57.816288 6132 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 03:57:57.817516 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:57.797945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.28.251.254:43295)
0808 03:57:57.798120 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:57.798125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:57.798537 (+ 412us) server_negotiation.cc:407] Connection header received
0808 03:57:57.798787 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:57.798792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:57.798884 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:57.799028 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0808 03:57:57.799569 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:57.800319 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:57.801724 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:57.802204 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:57.802324 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:57.802866 (+ 542us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 03:57:57.816778 (+ 13912us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 03:57:57.817191 (+ 413us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39198: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 03:57:57.817664 6090 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39198: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 03:57:57.817755 6131 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:57:57.797756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43295 (local address 127.0.0.1:39198)
0808 03:57:57.798420 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:57.798439 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:57.798598 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:57.799068 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:57.799078 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:57.799091 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0808 03:57:57.799387 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:57.799397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:57.800477 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:57.800483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:57.801568 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:57.801581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:57.802388 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:57.802395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:57.802548 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:57.802585 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 03:57:57.817061 (+ 14476us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 03:57:57.817352 (+ 291us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.251.254:43295: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":353,"threads_started":1}
I20260808 03:57:57.819515 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 5987
I20260808 03:57:57.820785 6076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:57:57.918744 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 5987
2026-08-08T03:57:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (802 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 03:57:57.986536 29679 mini_oidc.cc:152] Starting JWKS server
I20260808 03:57:57.988698 29679 webserver.cc:460] Webserver started at https://127.0.0.1:43043/ using document root <none> and password file <none>
I20260808 03:57:57.988976 29679 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 03:57:57.989708 29679 webserver.cc:460] Webserver started at http://127.0.0.1:43487/ using document root <none> and password file <none>
2026-08-08T03:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:58Z Disabled control of system clock
I20260808 03:57:58.030542 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:44845
--webserver_interface=127.28.251.254
--webserver_port=0
--builtin_ntp_servers=127.28.251.212:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:44845
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43043/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 03:57:58.387490 6148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:58.387854 6148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:58.387995 6148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:58.395668 6148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:58.395813 6148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:58.395881 6148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:58.395961 6148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:58.396021 6148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 03:57:58.396118 6148 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 03:57:58.396173 6148 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43043/jwks.json
I20260808 03:57:58.405931 6148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:44845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:44845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43043/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:58.407358 6148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:58.409277 6148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:58.421881 6156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:58.421895 6155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:58.421766 6158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:58.422904 6148 server_base.cc:1034] running on GCE node
I20260808 03:57:58.423842 6148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:58.425405 6148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:58.426625 6148 hybrid_clock.cc:648] HybridClock initialized: now 1786161478426576 us; error 48 us; skew 500 ppm
I20260808 03:57:58.426997 6148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:57:58.429603 6148 webserver.cc:460] Webserver started at http://127.28.251.254:36717/ using document root <none> and password file <none>
I20260808 03:57:58.430388 6148 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:58.430508 6148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:58.430848 6148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:58.434100 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "cab4d50013ba4d5094e55e87722f8670"
format_stamp: "Formatted at 2026-08-08 03:57:58 on dist-test-slave-pww0"
server_key: "8614dac86c35bd7419ff713dd56abf2e"
server_key_iv: "2e4ffce9e692a650bb09a0b3be613a36"
server_key_version: "encryptionkey@0"
I20260808 03:57:58.435027 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "cab4d50013ba4d5094e55e87722f8670"
format_stamp: "Formatted at 2026-08-08 03:57:58 on dist-test-slave-pww0"
server_key: "8614dac86c35bd7419ff713dd56abf2e"
server_key_iv: "2e4ffce9e692a650bb09a0b3be613a36"
server_key_version: "encryptionkey@0"
I20260808 03:57:58.440834 6148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 03:57:58.444866 6174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:58.446341 6148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:57:58.446537 6148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "cab4d50013ba4d5094e55e87722f8670"
format_stamp: "Formatted at 2026-08-08 03:57:58 on dist-test-slave-pww0"
server_key: "8614dac86c35bd7419ff713dd56abf2e"
server_key_iv: "2e4ffce9e692a650bb09a0b3be613a36"
server_key_version: "encryptionkey@0"
I20260808 03:57:58.446765 6148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:58.458922 6148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:58.459942 6148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:58.460258 6148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:58.479228 6148 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:44845
I20260808 03:57:58.479240 6254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:44845 every 8 connection(s)
I20260808 03:57:58.480738 6148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:58.484822 6256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:58.488106 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 6148
I20260808 03:57:58.488355 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:58.489198 29679 external_mini_cluster.cc:1428] Setting key ac3ef0e2461f975e33d55b17ff409504
I20260808 03:57:58.498049 6256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670: Bootstrap starting.
I20260808 03:57:58.501482 6256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:58.502693 6256 log.cc:824] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:58.503863 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:58.491449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.28.251.254:44845)
0808 03:57:58.492607 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:58.492619 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:58.492678 (+ 59us) server_negotiation.cc:407] Connection header received
0808 03:57:58.494426 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:58.494454 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:58.495030 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:58.495498 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:58.496145 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:58.497467 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:58.498883 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:58.499674 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:58.499863 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:58.501033 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:58.501041 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:58.501058 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:58.501679 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:58.501775 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:58.501799 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:58.501912 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:58.502159 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 03:57:58.502483 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":383,"threads_started":1}
I20260808 03:57:58.506233 6256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670: No bootstrap required, opened a new log
I20260808 03:57:58.508123 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 03:57:58.514329 6256 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab4d50013ba4d5094e55e87722f8670" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44845 } }
I20260808 03:57:58.514789 6256 raft_consensus.cc:374] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:58.514906 6256 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab4d50013ba4d5094e55e87722f8670, State: Initialized, Role: FOLLOWER
I20260808 03:57:58.515559 6256 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab4d50013ba4d5094e55e87722f8670" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44845 } }
I20260808 03:57:58.515813 6256 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:58.515945 6256 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:58.516109 6256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:58.518582 6256 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab4d50013ba4d5094e55e87722f8670" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44845 } }
I20260808 03:57:58.519131 6256 leader_election.cc:302] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab4d50013ba4d5094e55e87722f8670; no voters:
I20260808 03:57:58.519781 6256 leader_election.cc:288] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:58.519922 6265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:58.521394 6265 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [term 1 LEADER]: Becoming Leader. State: Replica: cab4d50013ba4d5094e55e87722f8670, State: Running, Role: LEADER
I20260808 03:57:58.522172 6256 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:58.522059 6265 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab4d50013ba4d5094e55e87722f8670" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44845 } }
I20260808 03:57:58.524809 6267 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab4d50013ba4d5094e55e87722f8670. Latest consensus state: current_term: 1 leader_uuid: "cab4d50013ba4d5094e55e87722f8670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab4d50013ba4d5094e55e87722f8670" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44845 } } }
I20260808 03:57:58.525671 6267 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:58.525951 6266 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cab4d50013ba4d5094e55e87722f8670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab4d50013ba4d5094e55e87722f8670" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44845 } } }
I20260808 03:57:58.526351 6266 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670 [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:58.528047 6275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:58.533201 6275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:58.543110 6275 catalog_manager.cc:1347] Generated new cluster ID: dee418a088a6488cba891279910cdcca
I20260808 03:57:58.543267 6275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:58.563481 6275 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:58.565044 6275 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:58.572080 6275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cab4d50013ba4d5094e55e87722f8670: Generated new TSK 0
I20260808 03:57:58.572980 6275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 03:57:58.598644 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:58.589030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.28.251.254:44845)
0808 03:57:58.589270 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:58.589279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:58.589784 (+ 505us) server_negotiation.cc:407] Connection header received
0808 03:57:58.590048 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:58.590052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:58.590129 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:58.590284 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0808 03:57:58.590932 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:58.591753 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:58.593864 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:58.594347 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:58.594457 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:58.595204 (+ 747us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 03:57:58.597923 (+ 2719us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43043/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 03:57:58.598314 (+ 391us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33612: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43043/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 03:57:58.598778 6262 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33612: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43043/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 03:57:58.600903 6305 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:57:58.588996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:44845 (local address 127.0.0.1:33612)
0808 03:57:58.589656 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:58.589679 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:58.589865 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:58.590333 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:58.590339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:58.590363 (+ 24us) client_negotiation.cc:189] Negotiated authn=JWT
0808 03:57:58.590766 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:58.590781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:58.591935 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:58.591941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:58.593668 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:58.593685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:58.594573 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:58.594581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:58.594792 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:58.594845 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 03:57:58.598187 (+ 3342us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43043/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 03:57:58.600414 (+ 2227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.251.254:44845: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43043/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":1862,"thread_start_us":352,"threads_started":1}
I20260808 03:57:58.601938 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 6148
I20260808 03:57:58.603765 6248 generic_service.cc:196] Checking for leaks (request via RPC)
E20260808 03:57:58.613262 6306 webserver.cc:533] Webserver: sslize failed:
E20260808 03:57:58.613421 6306 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 03:57:58.699916 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 6148
2026-08-08T03:57:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (775 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T03:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:57:58Z Disabled control of system clock
I20260808 03:57:58.802894 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:44261
--webserver_interface=127.28.251.254
--webserver_port=0
--builtin_ntp_servers=127.28.251.212:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:44261
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/testchainca.cert with env {}
W20260808 03:57:59.164070 6312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:57:59.164407 6312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:57:59.164544 6312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:57:59.172156 6312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:57:59.172289 6312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:57:59.172333 6312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:57:59.172415 6312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:57:59.172467 6312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 03:57:59.172557 6312 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 03:57:59.172605 6312 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 03:57:59.183252 6312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:44261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:44261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:57:59.184720 6312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:57:59.186124 6312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:57:59.198581 6319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:59.198786 6323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:57:59.198611 6320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:57:59.199383 6312 server_base.cc:1034] running on GCE node
I20260808 03:57:59.200428 6312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:57:59.201982 6312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:57:59.203222 6312 hybrid_clock.cc:648] HybridClock initialized: now 1786161479203169 us; error 46 us; skew 500 ppm
I20260808 03:57:59.203667 6312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:57:59.206225 6312 webserver.cc:460] Webserver started at http://127.28.251.254:40633/ using document root <none> and password file <none>
I20260808 03:57:59.207000 6312 fs_manager.cc:359] Metadata directory not provided
I20260808 03:57:59.207121 6312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:57:59.207444 6312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:57:59.210738 6312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "e53e002f087849a9a1f759ab39fe0add"
format_stamp: "Formatted at 2026-08-08 03:57:59 on dist-test-slave-pww0"
server_key: "666844d3920497465e63b1bd97304032"
server_key_iv: "b022c5fadd5b152aef5d90fabbef0c24"
server_key_version: "encryptionkey@0"
I20260808 03:57:59.211666 6312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "e53e002f087849a9a1f759ab39fe0add"
format_stamp: "Formatted at 2026-08-08 03:57:59 on dist-test-slave-pww0"
server_key: "666844d3920497465e63b1bd97304032"
server_key_iv: "b022c5fadd5b152aef5d90fabbef0c24"
server_key_version: "encryptionkey@0"
I20260808 03:57:59.217244 6312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 03:57:59.221206 6332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:57:59.222879 6312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 03:57:59.223124 6312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "e53e002f087849a9a1f759ab39fe0add"
format_stamp: "Formatted at 2026-08-08 03:57:59 on dist-test-slave-pww0"
server_key: "666844d3920497465e63b1bd97304032"
server_key_iv: "b022c5fadd5b152aef5d90fabbef0c24"
server_key_version: "encryptionkey@0"
I20260808 03:57:59.223385 6312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:57:59.248905 6312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:57:59.249943 6312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:57:59.250324 6312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:57:59.270206 6312 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:44261
I20260808 03:57:59.270227 6403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:44261 every 8 connection(s)
I20260808 03:57:59.271865 6312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:57:59.276054 6404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:57:59.281774 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 6312
I20260808 03:57:59.281985 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:57:59.282804 29679 external_mini_cluster.cc:1428] Setting key 4c426ef9b82ebd6c74499b97bd1a6a18
I20260808 03:57:59.288475 6404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add: Bootstrap starting.
I20260808 03:57:59.292369 6404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add: Neither blocks nor log segments found. Creating new log.
I20260808 03:57:59.293663 6404 log.cc:824] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add: Log is configured to *not* fsync() on all Append() calls
I20260808 03:57:59.297111 6404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add: No bootstrap required, opened a new log
I20260808 03:57:59.297685 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:59.285036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.28.251.254:44261)
0808 03:57:59.286185 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:59.286200 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:59.286250 (+ 50us) server_negotiation.cc:407] Connection header received
0808 03:57:59.288092 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:59.288118 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:59.288668 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:59.289040 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:57:59.289727 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:59.291111 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:59.292973 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:59.293719 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:59.293931 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:59.295236 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:57:59.295244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:57:59.295260 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:57:59.295868 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:57:59.295951 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:57:59.295974 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:57:59.296068 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 03:57:59.296300 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 03:57:59.296614 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":412,"threads_started":1}
I20260808 03:57:59.302268 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 03:57:59.307230 6404 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e002f087849a9a1f759ab39fe0add" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44261 } }
I20260808 03:57:59.307701 6404 raft_consensus.cc:374] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:57:59.307790 6404 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53e002f087849a9a1f759ab39fe0add, State: Initialized, Role: FOLLOWER
I20260808 03:57:59.308327 6404 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e002f087849a9a1f759ab39fe0add" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44261 } }
I20260808 03:57:59.308543 6404 raft_consensus.cc:388] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:57:59.308653 6404 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:57:59.308789 6404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:57:59.311216 6404 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e002f087849a9a1f759ab39fe0add" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44261 } }
I20260808 03:57:59.311899 6404 leader_election.cc:302] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53e002f087849a9a1f759ab39fe0add; no voters:
I20260808 03:57:59.312506 6404 leader_election.cc:288] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:57:59.312671 6414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:57:59.313616 6414 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [term 1 LEADER]: Becoming Leader. State: Replica: e53e002f087849a9a1f759ab39fe0add, State: Running, Role: LEADER
I20260808 03:57:59.314248 6414 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e002f087849a9a1f759ab39fe0add" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44261 } }
I20260808 03:57:59.314976 6404 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:57:59.316998 6415 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e53e002f087849a9a1f759ab39fe0add" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e002f087849a9a1f759ab39fe0add" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44261 } } }
I20260808 03:57:59.317400 6415 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:59.318042 6416 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53e002f087849a9a1f759ab39fe0add. Latest consensus state: current_term: 1 leader_uuid: "e53e002f087849a9a1f759ab39fe0add" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53e002f087849a9a1f759ab39fe0add" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 44261 } } }
I20260808 03:57:59.318432 6416 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add [sys.catalog]: This master's current role is: LEADER
I20260808 03:57:59.319320 6420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:57:59.325729 6420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:57:59.338217 6420 catalog_manager.cc:1347] Generated new cluster ID: 281cba6613a442478ef78b2393772398
I20260808 03:57:59.338378 6420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:57:59.358652 6420 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:57:59.360157 6420 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:57:59.367667 6420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e53e002f087849a9a1f759ab39fe0add: Generated new TSK 0
I20260808 03:57:59.368918 6420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 03:58:00.216800 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:57:59.392927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.28.251.254:44261)
0808 03:57:59.393122 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 03:57:59.393128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:57:59.393669 (+ 541us) server_negotiation.cc:407] Connection header received
0808 03:57:59.393915 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:57:59.393921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:57:59.394023 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:57:59.394220 (+ 197us) server_negotiation.cc:226] Negotiated authn=JWT
0808 03:57:59.394753 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:59.395944 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:59.397421 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:57:59.398108 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:57:59.398248 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:59.398754 (+ 506us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 03:58:00.216027 (+817273us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 03:58:00.216492 (+ 465us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34336: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 03:58:00.216913 6407 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34336: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 03:58:00.217240 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:57:59.392908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:44261 (local address 127.0.0.1:34336)
0808 03:57:59.393550 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0808 03:57:59.393568 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:57:59.393710 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:57:59.394269 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:57:59.394273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:57:59.394285 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0808 03:57:59.394560 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:59.394568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:59.396122 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:59.396127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:59.397259 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:57:59.397272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:57:59.398273 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:57:59.398278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:57:59.398401 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:57:59.398437 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 03:58:00.216352 (+817915us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 03:58:00.216666 (+ 314us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.251.254:44261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":328,"threads_started":1}
I20260808 03:58:00.218351 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 6312
I20260808 03:58:00.219853 6396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:00.325656 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 6312
2026-08-08T03:58:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1613 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (452 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (459 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:58:01 dist-test-slave-pww0 krb5kdc[6465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:58:01 dist-test-slave-pww0 krb5kdc[6465](info): set up 2 sockets
Aug 08 03:58:01 dist-test-slave-pww0 krb5kdc[6465](info): commencing operation
krb5kdc: starting...
W20260808 03:58:02.498653 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.004s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:58:02 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:58:02.780294 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:46525
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:46525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:03.134871 6485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:03.135195 6485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:03.135345 6485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:03.143077 6485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:58:03.143213 6485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:03.143267 6485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:03.143356 6485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:58:03.143412 6485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:58:03.154006 6485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:46525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:46525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:03.155553 6485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:03.157034 6485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:03.169410 6493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:03.169486 6494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:03.169368 6497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:03.170130 6485 server_base.cc:1034] running on GCE node
I20260808 03:58:03.171006 6485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:03.172720 6485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:03.173943 6485 hybrid_clock.cc:648] HybridClock initialized: now 1786161483173887 us; error 46 us; skew 500 ppm
I20260808 03:58:03.174328 6485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:03 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161483, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:03.180553 6485 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:58:03.182333 6485 webserver.cc:460] Webserver started at http://127.28.251.254:44737/ using document root <none> and password file <none>
I20260808 03:58:03.183089 6485 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:03.183207 6485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:03.183588 6485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:03.186866 6485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "093395387f4548beacdc0f273219c7af"
format_stamp: "Formatted at 2026-08-08 03:58:03 on dist-test-slave-pww0"
server_key: "e60effbbfdf52ee6774262d1b46aa9ba"
server_key_iv: "121a8a8f172fe6146ae9c5d41f3aeb12"
server_key_version: "encryptionkey@0"
I20260808 03:58:03.187812 6485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "093395387f4548beacdc0f273219c7af"
format_stamp: "Formatted at 2026-08-08 03:58:03 on dist-test-slave-pww0"
server_key: "e60effbbfdf52ee6774262d1b46aa9ba"
server_key_iv: "121a8a8f172fe6146ae9c5d41f3aeb12"
server_key_version: "encryptionkey@0"
I20260808 03:58:03.193794 6485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260808 03:58:03.198004 6505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:03.199672 6485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:03.199890 6485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "093395387f4548beacdc0f273219c7af"
format_stamp: "Formatted at 2026-08-08 03:58:03 on dist-test-slave-pww0"
server_key: "e60effbbfdf52ee6774262d1b46aa9ba"
server_key_iv: "121a8a8f172fe6146ae9c5d41f3aeb12"
server_key_version: "encryptionkey@0"
I20260808 03:58:03.200124 6485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:03.220950 6485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:03.225332 6485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:03.225620 6485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:03.244429 6485 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:46525
I20260808 03:58:03.244467 6583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:46525 every 8 connection(s)
I20260808 03:58:03.245944 6485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:58:03.248512 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 6485
I20260808 03:58:03.248718 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:58:03.249393 29679 external_mini_cluster.cc:1428] Setting key cc24d591d7df04cc5d6848fb9e408390
I20260808 03:58:03.250135 6584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 03:58:03 dist-test-slave-pww0 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:03.266270 6584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af: Bootstrap starting.
I20260808 03:58:03.270296 6584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:03.271834 6584 log.cc:824] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:03.275122 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:03.251554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.28.251.254:46525)
0808 03:58:03.252430 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:03.252447 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:03.252501 (+ 54us) server_negotiation.cc:407] Connection header received
0808 03:58:03.254390 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:03.254420 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:03.254886 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:03.255272 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:03.257243 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:03.258666 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:03.260438 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:03.261299 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:03.261472 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:03.266575 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:03.266586 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:03.266603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:03.266651 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:03.270138 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:03.271209 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:03.271217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:03.271226 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:03.271436 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:03.272007 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:03.272013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:03.272018 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:03.272878 (+ 860us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:03.273227 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:03.273620 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 03:58:03.274058 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":340,"threads_started":1}
I20260808 03:58:03.275442 6584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af: No bootstrap required, opened a new log
I20260808 03:58:03.284228 6584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093395387f4548beacdc0f273219c7af" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 46525 } }
I20260808 03:58:03.284655 6584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:03.284778 6584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093395387f4548beacdc0f273219c7af, State: Initialized, Role: FOLLOWER
I20260808 03:58:03.285387 6584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093395387f4548beacdc0f273219c7af" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 46525 } }
I20260808 03:58:03.285610 6584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:58:03.285758 6584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:58:03.285928 6584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:03.288563 6584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093395387f4548beacdc0f273219c7af" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 46525 } }
I20260808 03:58:03.289129 6584 leader_election.cc:302] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 093395387f4548beacdc0f273219c7af; no voters:
I20260808 03:58:03.289781 6584 leader_election.cc:288] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:58:03.289918 6590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:03.290593 6590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [term 1 LEADER]: Becoming Leader. State: Replica: 093395387f4548beacdc0f273219c7af, State: Running, Role: LEADER
I20260808 03:58:03.291208 6590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093395387f4548beacdc0f273219c7af" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 46525 } }
I20260808 03:58:03.292083 6584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:58:03.294468 6592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "093395387f4548beacdc0f273219c7af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093395387f4548beacdc0f273219c7af" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 46525 } } }
I20260808 03:58:03.294536 6593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 093395387f4548beacdc0f273219c7af. Latest consensus state: current_term: 1 leader_uuid: "093395387f4548beacdc0f273219c7af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093395387f4548beacdc0f273219c7af" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 46525 } } }
I20260808 03:58:03.294966 6592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:03.295034 6593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:03.297886 6603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:58:03.304247 6603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:58:03.314800 6603 catalog_manager.cc:1347] Generated new cluster ID: 37fec156b2614027a527ce59d616a3cf
I20260808 03:58:03.314935 6603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:58:03.327610 6603 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:58:03.329026 6603 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:58:03.336823 6603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 093395387f4548beacdc0f273219c7af: Generated new TSK 0
I20260808 03:58:03.338037 6603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:58:03.406724 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:46525
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:03.765013 6625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:03.765341 6625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:03.765457 6625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:03.773267 6625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:03.773408 6625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:03.773589 6625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:58:03.784421 6625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:46525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:03.785851 6625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:03.787312 6625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:03.801837 6635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:03.801712 6634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:03.801795 6637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:03.802567 6625 server_base.cc:1034] running on GCE node
I20260808 03:58:03.803231 6625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:03.804399 6625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:03.805639 6625 hybrid_clock.cc:648] HybridClock initialized: now 1786161483805570 us; error 59 us; skew 500 ppm
I20260808 03:58:03.805992 6625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:03 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161483, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:03.809240 6625 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:58:03.810894 6625 webserver.cc:460] Webserver started at http://127.28.251.193:37497/ using document root <none> and password file <none>
I20260808 03:58:03.811702 6625 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:03.811836 6625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:03.812156 6625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:03.815243 6625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "e090bcdaeeea4229b9ff2eb36609e638"
format_stamp: "Formatted at 2026-08-08 03:58:03 on dist-test-slave-pww0"
server_key: "3d0cf99dbb142071430ed0e43a922ad6"
server_key_iv: "bdad2be1383c91497911bddbe3a2ae0e"
server_key_version: "encryptionkey@0"
I20260808 03:58:03.816154 6625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "e090bcdaeeea4229b9ff2eb36609e638"
format_stamp: "Formatted at 2026-08-08 03:58:03 on dist-test-slave-pww0"
server_key: "3d0cf99dbb142071430ed0e43a922ad6"
server_key_iv: "bdad2be1383c91497911bddbe3a2ae0e"
server_key_version: "encryptionkey@0"
I20260808 03:58:03.821772 6625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 03:58:03.825637 6647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:03.827093 6625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 03:58:03.827276 6625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "e090bcdaeeea4229b9ff2eb36609e638"
format_stamp: "Formatted at 2026-08-08 03:58:03 on dist-test-slave-pww0"
server_key: "3d0cf99dbb142071430ed0e43a922ad6"
server_key_iv: "bdad2be1383c91497911bddbe3a2ae0e"
server_key_version: "encryptionkey@0"
I20260808 03:58:03.827538 6625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:03.843417 6625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:03.847892 6625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:03.848189 6625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:03.849254 6625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:03.850716 6625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:03.850800 6625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:03.850916 6625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:03.850975 6625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:03.884572 6625 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:33033
I20260808 03:58:03.884649 6849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:33033 every 8 connection(s)
I20260808 03:58:03.885932 6625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:58:03.895666 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 6625
I20260808 03:58:03.895884 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:58:03.896763 29679 external_mini_cluster.cc:1428] Setting key 1726d3b7913e0a5b6924face10b800fc
Aug 08 03:58:03 dist-test-slave-pww0 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161483, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:03.908089 6857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:03.888634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:42453 (local address 127.28.251.254:46525)
0808 03:58:03.889111 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:03.889117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:03.890265 (+ 1148us) server_negotiation.cc:407] Connection header received
0808 03:58:03.892163 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:03.892168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:03.892259 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:03.892357 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:03.894034 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:03.894734 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:03.897145 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:03.897551 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:03.897677 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:03.901679 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:03.901685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:03.901688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:03.901726 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:03.904710 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:03.905594 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:03.905600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:03.905603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:03.905766 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:03.906205 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:03.906209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:03.906212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:03.906572 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:03.906717 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:03.907489 (+ 772us) server_negotiation.cc:299] Negotiation successful
0808 03:58:03.907734 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":217,"threads_started":1}
I20260808 03:58:03.909057 6858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:03.889252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:46525 (local address 127.28.251.193:42453)
0808 03:58:03.890068 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:03.890116 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:03.891904 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:03.892593 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:03.892602 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:03.893206 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:03.893864 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:03.893884 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:03.894906 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:03.894911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:03.896991 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:03.897009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:03.897787 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:03.897793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:03.898073 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:03.898877 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:03.898902 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:03.901434 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:03.904880 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:03.904888 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:03.904906 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:03.905458 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:03.905870 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:03.905874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:03.905876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:03.906087 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:03.906689 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:03.906696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:03.907068 (+ 372us) client_negotiation.cc:769] Sending connection context
0808 03:58:03.907309 (+ 241us) client_negotiation.cc:240] Negotiation successful
0808 03:58:03.907809 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":324,"threads_started":1}
I20260808 03:58:03.910720 6852 heartbeater.cc:347] Connected to a master server at 127.28.251.254:46525
I20260808 03:58:03.911110 6852 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:03.912119 6852 heartbeater.cc:511] Master 127.28.251.254:46525 requested a full tablet report, sending...
I20260808 03:58:03.914402 6531 ts_manager.cc:194] Registered new tserver with Master: e090bcdaeeea4229b9ff2eb36609e638 (127.28.251.193:33033)
I20260808 03:58:03.915951 6531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:42453
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:58:03.954628 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:46525
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:04.310372 6862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:04.310710 6862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:04.310869 6862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:04.318435 6862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:04.318586 6862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:04.318748 6862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:58:04.329066 6862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:46525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:04.330611 6862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:04.332119 6862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:04.345985 6877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:04.345957 6880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:04.345973 6871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:04.346884 6862 server_base.cc:1034] running on GCE node
I20260808 03:58:04.347589 6862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:04.348626 6862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:04.349838 6862 hybrid_clock.cc:648] HybridClock initialized: now 1786161484349783 us; error 45 us; skew 500 ppm
I20260808 03:58:04.350155 6862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:04 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161484, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:04.353471 6862 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:58:04.355087 6862 webserver.cc:460] Webserver started at http://127.28.251.194:46443/ using document root <none> and password file <none>
I20260808 03:58:04.355892 6862 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:04.356024 6862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:04.356348 6862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:04.359601 6862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "f8f46ac4445e400aa6ccfced1cc2799b"
format_stamp: "Formatted at 2026-08-08 03:58:04 on dist-test-slave-pww0"
server_key: "b434eb79366617be92ab3f1dfeb7a70c"
server_key_iv: "5125df9fd3c84d1785f4833470e45076"
server_key_version: "encryptionkey@0"
I20260808 03:58:04.360466 6862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "f8f46ac4445e400aa6ccfced1cc2799b"
format_stamp: "Formatted at 2026-08-08 03:58:04 on dist-test-slave-pww0"
server_key: "b434eb79366617be92ab3f1dfeb7a70c"
server_key_iv: "5125df9fd3c84d1785f4833470e45076"
server_key_version: "encryptionkey@0"
I20260808 03:58:04.366302 6862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260808 03:58:04.370288 6895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:04.371853 6862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:04.372037 6862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "f8f46ac4445e400aa6ccfced1cc2799b"
format_stamp: "Formatted at 2026-08-08 03:58:04 on dist-test-slave-pww0"
server_key: "b434eb79366617be92ab3f1dfeb7a70c"
server_key_iv: "5125df9fd3c84d1785f4833470e45076"
server_key_version: "encryptionkey@0"
I20260808 03:58:04.372259 6862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:04.390244 6862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:04.395150 6862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:04.395488 6862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:04.396674 6862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:04.398325 6862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:04.398412 6862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:04.398538 6862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:04.398599 6862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:04.431942 6862 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:35643
I20260808 03:58:04.432005 7077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:35643 every 8 connection(s)
I20260808 03:58:04.433420 6862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:58:04.443593 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 6862
I20260808 03:58:04.443794 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:58:04.444419 29679 external_mini_cluster.cc:1428] Setting key 9e1ec1531c4c3d94b8811537d49d8d26
Aug 08 03:58:04 dist-test-slave-pww0 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161484, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:04.456187 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:04.436184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:46955 (local address 127.28.251.254:46525)
0808 03:58:04.436656 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:04.436662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:04.437738 (+ 1076us) server_negotiation.cc:407] Connection header received
0808 03:58:04.439671 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:04.439675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:04.439761 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:04.439850 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:04.441530 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:04.442282 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:04.444570 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:04.445034 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:04.445161 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:04.449321 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:04.449328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:04.449332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:04.449385 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:04.452305 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:04.453466 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:04.453472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:04.453474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:04.453650 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:04.454265 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:04.454269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:04.454271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:04.454577 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:04.454698 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:04.455697 (+ 999us) server_negotiation.cc:299] Negotiation successful
0808 03:58:04.455898 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":231,"threads_started":1}
I20260808 03:58:04.457664 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:04.436790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:46525 (local address 127.28.251.194:46955)
0808 03:58:04.437558 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:04.437602 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:04.439341 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:04.440101 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:04.440111 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:04.440693 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:04.441338 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:04.441358 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:04.442459 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:04.442463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:04.444355 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:04.444375 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:04.445304 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:04.445312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:04.445604 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:04.446374 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:04.446404 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:04.448987 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:04.452521 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:04.452532 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:04.452554 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:04.453273 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:04.453771 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:04.453775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:04.453778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:04.454096 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:04.454709 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:04.454718 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:04.455234 (+ 516us) client_negotiation.cc:769] Sending connection context
0808 03:58:04.455605 (+ 371us) client_negotiation.cc:240] Negotiation successful
0808 03:58:04.456208 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":291,"threads_started":1}
I20260808 03:58:04.459437 7079 heartbeater.cc:347] Connected to a master server at 127.28.251.254:46525
I20260808 03:58:04.459808 7079 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:04.460610 7079 heartbeater.cc:511] Master 127.28.251.254:46525 requested a full tablet report, sending...
I20260808 03:58:04.462169 6531 ts_manager.cc:194] Registered new tserver with Master: f8f46ac4445e400aa6ccfced1cc2799b (127.28.251.194:35643)
I20260808 03:58:04.463033 6531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:46955
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:58:04.504541 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:46525
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:04.859238 7090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:04.859659 7090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:04.859820 7090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:04.867825 7090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:04.867965 7090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:04.868141 7090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:58:04.879017 7090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:46525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:04.880525 7090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:04.881906 7090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:04.895749 7100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:04.895792 7099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:04.895689 7104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:04.896515 7090 server_base.cc:1034] running on GCE node
I20260808 03:58:04.897189 7090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:04.898205 7090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:04.899415 7090 hybrid_clock.cc:648] HybridClock initialized: now 1786161484899349 us; error 55 us; skew 500 ppm
I20260808 03:58:04.899820 7090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:04 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161484, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:04.903133 7090 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:58:04.904775 7090 webserver.cc:460] Webserver started at http://127.28.251.195:41327/ using document root <none> and password file <none>
I20260808 03:58:04.905524 7090 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:04.905637 7090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:04.905920 7090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:04.909159 7090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "007e9b8357474ffda4ca43fcb011cfb4"
format_stamp: "Formatted at 2026-08-08 03:58:04 on dist-test-slave-pww0"
server_key: "c4b32b832d1f59df28e58f594b0cb3bc"
server_key_iv: "6bd83e481203d7b9367a801ce5e79393"
server_key_version: "encryptionkey@0"
I20260808 03:58:04.909991 7090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "007e9b8357474ffda4ca43fcb011cfb4"
format_stamp: "Formatted at 2026-08-08 03:58:04 on dist-test-slave-pww0"
server_key: "c4b32b832d1f59df28e58f594b0cb3bc"
server_key_iv: "6bd83e481203d7b9367a801ce5e79393"
server_key_version: "encryptionkey@0"
I20260808 03:58:04.915897 7090 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 03:58:04.919205 6852 heartbeater.cc:503] Master 127.28.251.254:46525 was elected leader, sending a full tablet report...
I20260808 03:58:04.919979 7116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:04.921715 7090 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 03:58:04.921875 7090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "007e9b8357474ffda4ca43fcb011cfb4"
format_stamp: "Formatted at 2026-08-08 03:58:04 on dist-test-slave-pww0"
server_key: "c4b32b832d1f59df28e58f594b0cb3bc"
server_key_iv: "6bd83e481203d7b9367a801ce5e79393"
server_key_version: "encryptionkey@0"
I20260808 03:58:04.922072 7090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:04.947224 7090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:04.952279 7090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:04.952565 7090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:04.953639 7090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:04.955084 7090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:04.955170 7090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:04.955271 7090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:04.955314 7090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:04.987622 7090 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:46867
I20260808 03:58:04.987680 7291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:46867 every 8 connection(s)
I20260808 03:58:04.989004 7090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:58:04.992720 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 7090
I20260808 03:58:04.992938 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:58:04.993577 29679 external_mini_cluster.cc:1428] Setting key ee9901a9073573f502cfa57361269996
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161484, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:05.011201 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:04.991775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:40539 (local address 127.28.251.254:46525)
0808 03:58:04.992338 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:04.992345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:04.993610 (+ 1265us) server_negotiation.cc:407] Connection header received
0808 03:58:04.996306 (+ 2696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:04.996312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:04.996440 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:04.996559 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:04.998295 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:04.999176 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.000910 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.001335 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.001435 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:05.005183 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:05.005189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:05.005192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:05.005229 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:05.007806 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.008760 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.008765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.008767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.008940 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.009488 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.009493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.009495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.009853 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:05.009978 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:05.010718 (+ 740us) server_negotiation.cc:299] Negotiation successful
0808 03:58:05.010908 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":240,"threads_started":1}
I20260808 03:58:05.012260 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:04.992426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:46525 (local address 127.28.251.195:40539)
0808 03:58:04.993420 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:04.993477 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:04.996008 (+ 2531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:04.996832 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:04.996845 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:04.997516 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:04.998138 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:04.998156 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:04.999365 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:04.999369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:05.000742 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:05.000756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.001561 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.001567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:05.001788 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:05.002464 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:05.002488 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:05.004952 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:05.007953 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:05.007963 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:05.007983 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:05.008632 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:05.009070 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:05.009075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:05.009078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:05.009359 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:05.009985 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:05.009991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:05.010376 (+ 385us) client_negotiation.cc:769] Sending connection context
0808 03:58:05.010626 (+ 250us) client_negotiation.cc:240] Negotiation successful
0808 03:58:05.011057 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":368,"threads_started":1}
I20260808 03:58:05.013710 7293 heartbeater.cc:347] Connected to a master server at 127.28.251.254:46525
I20260808 03:58:05.014053 7293 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:05.014798 7293 heartbeater.cc:511] Master 127.28.251.254:46525 requested a full tablet report, sending...
I20260808 03:58:05.016285 6531 ts_manager.cc:194] Registered new tserver with Master: 007e9b8357474ffda4ca43fcb011cfb4 (127.28.251.195:46867)
I20260808 03:58:05.017117 6531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:40539
I20260808 03:58:05.021910 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/test-admin.keytab.
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:05.055426 29679 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 03:58:05.061175 7308 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:05.058298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.28.251.193:33033)
0808 03:58:05.058980 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:05.058990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:05.059028 (+ 38us) server_negotiation.cc:407] Connection header received
0808 03:58:05.059110 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:05.059119 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:05.059343 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:05.059529 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:05.060717 (+ 1188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57434: BlockingRecv error: recv got EOF from 127.0.0.1:57434 (error 108)
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":303,"threads_started":1}
W20260808 03:58:05.061358 29679 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/krb5cc' of type FILE
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:05.062937 29679 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:05.064417 29679 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:58:05.081557 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:05.066238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.28.251.193:33033)
0808 03:58:05.066459 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:05.066465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:05.066693 (+ 228us) server_negotiation.cc:407] Connection header received
0808 03:58:05.066902 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:05.066907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:05.066972 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:05.067058 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:05.068222 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.069269 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.070555 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.071174 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.071299 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:05.075093 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:05.075110 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:05.075128 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:05.075178 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:05.078472 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.079289 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.079297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.079300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.079448 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.079898 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.079902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.079904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.080317 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:05.080622 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:05.080992 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 03:58:05.081274 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 03:58:05.087743 7308 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:05.085899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.28.251.193:33033)
0808 03:58:05.086115 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:05.086121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:05.086253 (+ 132us) server_negotiation.cc:407] Connection header received
0808 03:58:05.086438 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:05.086444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:05.086526 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:05.086608 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:05.087488 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57450: BlockingRecv error: recv got EOF from 127.0.0.1:57450 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 03:58:05.087881 29679 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/krb5cc' of type FILE
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:05.089435 29679 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:05.090870 29679 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:58:05.106006 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:05.092387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.28.251.193:33033)
0808 03:58:05.092601 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:05.092606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:05.092805 (+ 199us) server_negotiation.cc:407] Connection header received
0808 03:58:05.092986 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:05.092992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:05.093058 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:05.093144 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:05.094184 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.094965 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.096268 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.096718 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.096865 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:05.100653 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:05.100659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:05.100663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:05.100705 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:05.103171 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.104020 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.104026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.104028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.104169 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.104596 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.104599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.104602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.104947 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:05.105213 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:05.105515 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 03:58:05.105730 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 03:58:05.110272 7308 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:05.108709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.28.251.193:33033)
0808 03:58:05.108928 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:05.108934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:05.109122 (+ 188us) server_negotiation.cc:407] Connection header received
0808 03:58:05.109244 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:05.109248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:05.109325 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:05.109404 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:05.110053 (+ 649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57464: BlockingRecv error: recv got EOF from 127.0.0.1:57464 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 03:58:05.110482 29679 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786161430375636-29679-0/krb5kdc/krb5cc' of type FILE
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:05.112046 29679 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:05.113322 29679 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:58:05.128566 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:05.114660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.28.251.193:33033)
0808 03:58:05.114835 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:05.114840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:05.115191 (+ 351us) server_negotiation.cc:407] Connection header received
0808 03:58:05.115366 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:05.115370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:05.115432 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:05.115540 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:05.116552 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.117276 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.118538 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.118990 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.119103 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:05.122668 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:05.122674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:05.122677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:05.122717 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:05.125446 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.126170 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.126176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.126179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.126348 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.126771 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.126778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.126781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.127225 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:05.127594 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:05.127910 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 03:58:05.128179 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 03:58:05.130712 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 6625
I20260808 03:58:05.143378 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:05.131514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.28.251.193:33033)
0808 03:58:05.131695 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:05.131702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:05.132531 (+ 829us) server_negotiation.cc:407] Connection header received
0808 03:58:05.132773 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:05.132779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:05.132874 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:05.132964 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:05.134241 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.134932 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.136168 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.136598 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.136695 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:05.138625 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:05.138630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:05.138633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:05.138665 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:05.141166 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.141794 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.141797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.141800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.141951 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.142242 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.142245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.142247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.142503 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:05.142706 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:05.142945 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 03:58:05.143117 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 03:58:05.144075 6840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:05.231348 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 6625
I20260808 03:58:05.256724 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 6862
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 03:58:05.273092 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:05.257658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.28.251.194:35643)
0808 03:58:05.258258 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:05.258265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:05.258290 (+ 25us) server_negotiation.cc:407] Connection header received
0808 03:58:05.258359 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:05.258366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:05.258578 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:05.258733 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:05.259818 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.260897 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.262216 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.262781 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.262897 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:05.266933 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:05.266947 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:05.266963 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:05.267002 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:05.270001 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.270778 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.270787 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.270789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.270935 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.271330 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.271334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.271342 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.271795 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:05.272110 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:05.272482 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 03:58:05.272764 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":291,"threads_started":1}
I20260808 03:58:05.274488 7072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:05.358177 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 6862
I20260808 03:58:05.383038 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 7090
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 03:58:05.399969 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:05.384057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.28.251.195:46867)
0808 03:58:05.384663 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:05.384673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:05.384707 (+ 34us) server_negotiation.cc:407] Connection header received
0808 03:58:05.384775 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:05.384781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:05.384990 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:05.385148 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:05.386284 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.387382 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.388930 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:05.389547 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:05.389676 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:05.394001 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:05.394017 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:05.394033 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:05.394080 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:05.396944 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.397676 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.397684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.397686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.397826 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:05.398268 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:05.398272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:05.398274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:05.398670 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:05.398978 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:05.399306 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 03:58:05.399644 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":278,"threads_started":1}
I20260808 03:58:05.401343 7284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:05.486224 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 7090
I20260808 03:58:05.514418 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 6485
I20260808 03:58:05.515834 6579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:05.625412 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 6485
2026-08-08T03:58:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4385 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[7348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[7348](info): set up 2 sockets
Aug 08 03:58:05 dist-test-slave-pww0 krb5kdc[7348](info): commencing operation
krb5kdc: starting...
W20260808 03:58:06.880187 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.003s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:58:07 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:58:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:58:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.254.keytab.
Entry for principal oryx/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.254.keytab.
I20260808 03:58:07.175034 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40631
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.254.keytab
--principal=oryx/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:40631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:07.534759 7371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:07.535146 7371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:07.535300 7371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:07.542905 7371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:58:07.543057 7371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:07.543140 7371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:07.543226 7371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:58:07.543288 7371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:58:07.553732 7371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:40631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.254.keytab
--principal=oryx/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:07.555281 7371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:07.556788 7371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:07.569353 7383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:07.569319 7386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:07.569383 7384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:07.569962 7371 server_base.cc:1034] running on GCE node
I20260808 03:58:07.570961 7371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:07.572608 7371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:07.573844 7371 hybrid_clock.cc:648] HybridClock initialized: now 1786161487573801 us; error 55 us; skew 500 ppm
I20260808 03:58:07.574230 7371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:07 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161487, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:07.577658 7371 init.cc:377] Logged in from keytab as oryx/127.28.251.254@KRBTEST.COM (short username oryx)
I20260808 03:58:07.579432 7371 webserver.cc:460] Webserver started at http://127.28.251.254:46493/ using document root <none> and password file <none>
I20260808 03:58:07.580292 7371 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:07.580428 7371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:07.580791 7371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:07.584220 7371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "7b56826f1bab45dfb469a8b01ca93516"
format_stamp: "Formatted at 2026-08-08 03:58:07 on dist-test-slave-pww0"
server_key: "72dc312dfa3ba64695106ed528adfb23"
server_key_iv: "39466d9dd62aab0c51914e7cb586d408"
server_key_version: "encryptionkey@0"
I20260808 03:58:07.585080 7371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "7b56826f1bab45dfb469a8b01ca93516"
format_stamp: "Formatted at 2026-08-08 03:58:07 on dist-test-slave-pww0"
server_key: "72dc312dfa3ba64695106ed528adfb23"
server_key_iv: "39466d9dd62aab0c51914e7cb586d408"
server_key_version: "encryptionkey@0"
I20260808 03:58:07.590821 7371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 03:58:07.594741 7401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:07.596287 7371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:07.596498 7371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "7b56826f1bab45dfb469a8b01ca93516"
format_stamp: "Formatted at 2026-08-08 03:58:07 on dist-test-slave-pww0"
server_key: "72dc312dfa3ba64695106ed528adfb23"
server_key_iv: "39466d9dd62aab0c51914e7cb586d408"
server_key_version: "encryptionkey@0"
I20260808 03:58:07.596714 7371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:07.624030 7371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:07.628829 7371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:07.629135 7371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:07.648296 7371 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:40631
I20260808 03:58:07.648329 7496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:40631 every 8 connection(s)
I20260808 03:58:07.649869 7371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:58:07.654207 7498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:07.655259 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 7371
I20260808 03:58:07.655522 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:58:07.656411 29679 external_mini_cluster.cc:1428] Setting key 58f61b07d0118c6cbf3a44ff0287d109
I20260808 03:58:07.670404 7498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516: Bootstrap starting.
Aug 08 03:58:07 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:07.674703 7498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:07.675997 7498 log.cc:824] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:07.679941 7498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516: No bootstrap required, opened a new log
I20260808 03:58:07.683274 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:07.658843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.28.251.254:40631)
0808 03:58:07.659827 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:07.659840 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:07.659895 (+ 55us) server_negotiation.cc:407] Connection header received
0808 03:58:07.661726 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:07.661756 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:07.662274 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:07.662709 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:07.664742 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:07.666175 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:07.668178 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:07.669005 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:07.669191 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:07.674332 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:07.674343 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:07.674361 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:07.674416 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:07.678118 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:07.679240 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:07.679250 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:07.679259 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:07.679520 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:07.680189 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:07.680213 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:07.680215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:07.680850 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:07.681116 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:07.681656 (+ 540us) server_negotiation.cc:299] Negotiation successful
0808 03:58:07.682066 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":450,"threads_started":1}
I20260808 03:58:07.688637 7498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } }
I20260808 03:58:07.689060 7498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:07.689178 7498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b56826f1bab45dfb469a8b01ca93516, State: Initialized, Role: FOLLOWER
I20260808 03:58:07.689769 7498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } }
I20260808 03:58:07.689997 7498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:58:07.690132 7498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:58:07.690315 7498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:07.692764 7498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } }
I20260808 03:58:07.693313 7498 leader_election.cc:302] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b56826f1bab45dfb469a8b01ca93516; no voters:
I20260808 03:58:07.693945 7498 leader_election.cc:288] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:58:07.694099 7507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:07.695129 7507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 1 LEADER]: Becoming Leader. State: Replica: 7b56826f1bab45dfb469a8b01ca93516, State: Running, Role: LEADER
I20260808 03:58:07.695868 7507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } }
I20260808 03:58:07.696297 7498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:58:07.698894 7510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } }
I20260808 03:58:07.699321 7510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:07.698873 7511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b56826f1bab45dfb469a8b01ca93516. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } }
I20260808 03:58:07.699955 7511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:07.702179 7516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:58:07.707190 7516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:58:07.718166 7516 catalog_manager.cc:1347] Generated new cluster ID: 66b7c2628b5a49eb9ab7cf57e481cb60
I20260808 03:58:07.718690 7516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:58:07.755401 7516 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:58:07.756875 7516 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:58:07.764134 7516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516: Generated new TSK 0
I20260808 03:58:07.765033 7516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab.
Entry for principal oryx/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab.
I20260808 03:58:07.841432 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40631
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab
--principal=oryx/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:08.210269 7533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:08.210677 7533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:08.210855 7533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:08.218673 7533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:08.218818 7533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:08.218945 7533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:58:08.229305 7533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab
--principal=oryx/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:08.230687 7533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:08.232221 7533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:08.246057 7539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:08.246059 7543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:08.246047 7540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:08.246928 7533 server_base.cc:1034] running on GCE node
I20260808 03:58:08.247695 7533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:08.248761 7533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:08.250006 7533 hybrid_clock.cc:648] HybridClock initialized: now 1786161488249954 us; error 40 us; skew 500 ppm
I20260808 03:58:08.250370 7533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:08 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161488, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:08.257042 7533 init.cc:377] Logged in from keytab as oryx/127.28.251.193@KRBTEST.COM (short username oryx)
I20260808 03:58:08.258692 7533 webserver.cc:460] Webserver started at http://127.28.251.193:34225/ using document root <none> and password file <none>
I20260808 03:58:08.259575 7533 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:08.259786 7533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:08.260139 7533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:08.263435 7533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "446ce0b231b841a2875981263d95cad2"
format_stamp: "Formatted at 2026-08-08 03:58:08 on dist-test-slave-pww0"
server_key: "7816e2283dcd4d2837424a7b4041b508"
server_key_iv: "193c59dafc3c41cd822923a24f0f7b0e"
server_key_version: "encryptionkey@0"
I20260808 03:58:08.264415 7533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "446ce0b231b841a2875981263d95cad2"
format_stamp: "Formatted at 2026-08-08 03:58:08 on dist-test-slave-pww0"
server_key: "7816e2283dcd4d2837424a7b4041b508"
server_key_iv: "193c59dafc3c41cd822923a24f0f7b0e"
server_key_version: "encryptionkey@0"
I20260808 03:58:08.270275 7533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260808 03:58:08.274255 7554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:08.275766 7533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:08.275944 7533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "446ce0b231b841a2875981263d95cad2"
format_stamp: "Formatted at 2026-08-08 03:58:08 on dist-test-slave-pww0"
server_key: "7816e2283dcd4d2837424a7b4041b508"
server_key_iv: "193c59dafc3c41cd822923a24f0f7b0e"
server_key_version: "encryptionkey@0"
I20260808 03:58:08.276152 7533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:08.292433 7533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:08.297281 7533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:08.297582 7533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:08.298658 7533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:08.300258 7533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:08.300349 7533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:08.300478 7533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:08.300566 7533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:08.333102 7533 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:43373
I20260808 03:58:08.333163 7745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:43373 every 8 connection(s)
I20260808 03:58:08.334515 7533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:58:08.343312 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 7533
I20260808 03:58:08.343572 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:58:08.344360 29679 external_mini_cluster.cc:1428] Setting key 523cc80217e767021d6860516a6b9f22
Aug 08 03:58:08 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161488, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:08.356630 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:08.337312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:33979 (local address 127.28.251.254:40631)
0808 03:58:08.337800 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:08.337805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:08.338836 (+ 1031us) server_negotiation.cc:407] Connection header received
0808 03:58:08.340836 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:08.340842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:08.340935 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:08.341035 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:08.342724 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:08.343825 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:08.345739 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:08.346324 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:08.346447 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:08.350451 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:08.350457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:08.350460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:08.350502 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:08.353166 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:08.354026 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:08.354031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:08.354033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:08.354173 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:08.354657 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:08.354661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:08.354663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:08.354972 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:08.355120 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:08.355988 (+ 868us) server_negotiation.cc:299] Negotiation successful
0808 03:58:08.356252 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":251,"threads_started":1}
I20260808 03:58:08.357708 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:08.337816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.193:33979)
0808 03:58:08.338643 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:08.338692 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:08.340522 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:08.341323 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:08.341333 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:08.341904 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:08.342533 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:08.342552 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:08.344018 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:08.344023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:08.345582 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:08.345599 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:08.346581 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:08.346590 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:08.346849 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:08.347706 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:08.347733 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:08.350153 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:08.353372 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:08.353382 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:08.353399 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:08.353885 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:08.354294 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:08.354299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:08.354302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:08.354535 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:08.355112 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:08.355121 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:08.355566 (+ 445us) client_negotiation.cc:769] Sending connection context
0808 03:58:08.355845 (+ 279us) client_negotiation.cc:240] Negotiation successful
0808 03:58:08.356479 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":336,"threads_started":1}
I20260808 03:58:08.359530 7746 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40631
I20260808 03:58:08.359894 7746 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:08.360847 7746 heartbeater.cc:511] Master 127.28.251.254:40631 requested a full tablet report, sending...
I20260808 03:58:08.363579 7436 ts_manager.cc:194] Registered new tserver with Master: 446ce0b231b841a2875981263d95cad2 (127.28.251.193:43373)
I20260808 03:58:08.365670 7436 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:33979
WARNING: no policy specified for oryx/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab.
Entry for principal oryx/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab.
I20260808 03:58:08.408454 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40631
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab
--principal=oryx/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:08.764448 7754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:08.764740 7754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:08.764866 7754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:08.772353 7754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:08.772485 7754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:08.772611 7754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:58:08.782851 7754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab
--principal=oryx/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:08.784346 7754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:08.785832 7754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:08.799814 7771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:08.799741 7775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:08.799796 7773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:08.800673 7754 server_base.cc:1034] running on GCE node
I20260808 03:58:08.801367 7754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:08.802367 7754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:08.803555 7754 hybrid_clock.cc:648] HybridClock initialized: now 1786161488803493 us; error 55 us; skew 500 ppm
I20260808 03:58:08.803891 7754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:08 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161488, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:08.807085 7754 init.cc:377] Logged in from keytab as oryx/127.28.251.194@KRBTEST.COM (short username oryx)
I20260808 03:58:08.808696 7754 webserver.cc:460] Webserver started at http://127.28.251.194:43011/ using document root <none> and password file <none>
I20260808 03:58:08.809446 7754 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:08.809577 7754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:08.809896 7754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:08.813162 7754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "429e1d2b69e242f08e551d2021e5cba5"
format_stamp: "Formatted at 2026-08-08 03:58:08 on dist-test-slave-pww0"
server_key: "e1a0774cf85a8eb0ae487f9df58ec641"
server_key_iv: "5012381c1eab97cce34734d0ac2751e7"
server_key_version: "encryptionkey@0"
I20260808 03:58:08.814054 7754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "429e1d2b69e242f08e551d2021e5cba5"
format_stamp: "Formatted at 2026-08-08 03:58:08 on dist-test-slave-pww0"
server_key: "e1a0774cf85a8eb0ae487f9df58ec641"
server_key_iv: "5012381c1eab97cce34734d0ac2751e7"
server_key_version: "encryptionkey@0"
I20260808 03:58:08.820076 7754 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 03:58:08.824132 7784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:08.825623 7754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260808 03:58:08.825804 7754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "429e1d2b69e242f08e551d2021e5cba5"
format_stamp: "Formatted at 2026-08-08 03:58:08 on dist-test-slave-pww0"
server_key: "e1a0774cf85a8eb0ae487f9df58ec641"
server_key_iv: "5012381c1eab97cce34734d0ac2751e7"
server_key_version: "encryptionkey@0"
I20260808 03:58:08.826031 7754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:08.841265 7754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:08.845474 7754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:08.845788 7754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:08.846944 7754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:08.848480 7754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:08.848569 7754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:08.848714 7754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:08.848768 7754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:08.883554 7754 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:44113
I20260808 03:58:08.883586 7972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:44113 every 8 connection(s)
I20260808 03:58:08.885073 7754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:58:08.887545 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 7754
I20260808 03:58:08.887729 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:58:08.888346 29679 external_mini_cluster.cc:1428] Setting key cb8a5d66d270a49a846255b7dfa4ec6b
Aug 08 03:58:08 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161488, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:08.907735 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:08.888033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:44205 (local address 127.28.251.254:40631)
0808 03:58:08.888579 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:08.888586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:08.889940 (+ 1354us) server_negotiation.cc:407] Connection header received
0808 03:58:08.891930 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:08.891936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:08.892023 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:08.892148 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:08.893693 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:08.894705 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:08.896527 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:08.897144 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:08.897293 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:08.901098 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:08.901106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:08.901110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:08.901157 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:08.904218 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:08.905051 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:08.905057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:08.905060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:08.905228 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:08.905720 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:08.905725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:08.905728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:08.906138 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:08.906281 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:08.907057 (+ 776us) server_negotiation.cc:299] Negotiation successful
0808 03:58:08.907299 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":267,"threads_started":1}
I20260808 03:58:08.908600 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:08.888628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.194:44205)
0808 03:58:08.889767 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:08.889815 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:08.891641 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:08.892352 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:08.892361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:08.892891 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:08.893506 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:08.893526 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:08.894898 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:08.894902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:08.896347 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:08.896364 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:08.897399 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:08.897406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:08.897657 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:08.898352 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:08.898377 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:08.900791 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:08.904402 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:08.904410 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:08.904427 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:08.904900 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:08.905334 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:08.905337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:08.905339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:08.905577 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:08.906261 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:08.906268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:08.906654 (+ 386us) client_negotiation.cc:769] Sending connection context
0808 03:58:08.906898 (+ 244us) client_negotiation.cc:240] Negotiation successful
0808 03:58:08.907346 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":622,"threads_started":1}
I20260808 03:58:08.910308 7973 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40631
I20260808 03:58:08.910661 7973 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:08.911649 7973 heartbeater.cc:511] Master 127.28.251.254:40631 requested a full tablet report, sending...
I20260808 03:58:08.913417 7436 ts_manager.cc:194] Registered new tserver with Master: 429e1d2b69e242f08e551d2021e5cba5 (127.28.251.194:44113)
I20260808 03:58:08.914333 7436 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:44205
WARNING: no policy specified for oryx/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab.
Entry for principal oryx/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab.
I20260808 03:58:08.950690 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40631
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab
--principal=oryx/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:09.314606 7983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:09.314935 7983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:09.315085 7983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:09.322615 7983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:09.322765 7983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:09.322921 7983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:58:09.333271 7983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab
--principal=oryx/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:09.334640 7983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:09.335999 7983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:09.349790 7989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:09.349725 7991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:09.349690 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:09.350653 7983 server_base.cc:1034] running on GCE node
I20260808 03:58:09.351334 7983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:09.352438 7983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:09.353649 7983 hybrid_clock.cc:648] HybridClock initialized: now 1786161489353600 us; error 42 us; skew 500 ppm
I20260808 03:58:09.353986 7983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:09 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161489, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:09.357371 7983 init.cc:377] Logged in from keytab as oryx/127.28.251.195@KRBTEST.COM (short username oryx)
I20260808 03:58:09.359005 7983 webserver.cc:460] Webserver started at http://127.28.251.195:34893/ using document root <none> and password file <none>
I20260808 03:58:09.359787 7983 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:09.359926 7983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:09.360257 7983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:09.363569 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "d37b8ac688c142a6b3b478912340a1e2"
format_stamp: "Formatted at 2026-08-08 03:58:09 on dist-test-slave-pww0"
server_key: "c186f16dbe595f1ce498786af3fa8466"
server_key_iv: "80fed0b7091c289e54e6123c3f54cdf5"
server_key_version: "encryptionkey@0"
I20260808 03:58:09.364444 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "d37b8ac688c142a6b3b478912340a1e2"
format_stamp: "Formatted at 2026-08-08 03:58:09 on dist-test-slave-pww0"
server_key: "c186f16dbe595f1ce498786af3fa8466"
server_key_iv: "80fed0b7091c289e54e6123c3f54cdf5"
server_key_version: "encryptionkey@0"
I20260808 03:58:09.369740 7746 heartbeater.cc:503] Master 127.28.251.254:40631 was elected leader, sending a full tablet report...
I20260808 03:58:09.370169 7983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 03:58:09.374249 8009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:09.375792 7983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 03:58:09.375981 7983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "d37b8ac688c142a6b3b478912340a1e2"
format_stamp: "Formatted at 2026-08-08 03:58:09 on dist-test-slave-pww0"
server_key: "c186f16dbe595f1ce498786af3fa8466"
server_key_iv: "80fed0b7091c289e54e6123c3f54cdf5"
server_key_version: "encryptionkey@0"
I20260808 03:58:09.376202 7983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:09.399406 7983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:09.404234 7983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:09.404547 7983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:09.405617 7983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:09.407083 7983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:09.407162 7983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:09.407284 7983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:09.407335 7983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:09.439343 7983 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:34591
I20260808 03:58:09.439421 8192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:34591 every 8 connection(s)
I20260808 03:58:09.440740 7983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:58:09.449715 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 7983
I20260808 03:58:09.449920 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:58:09.450381 29679 external_mini_cluster.cc:1428] Setting key ebacdb4794737536ceb25240d9d0ae4c
Aug 08 03:58:09 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161489, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:09.461872 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:09.443391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:50997 (local address 127.28.251.254:40631)
0808 03:58:09.443895 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:09.443900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:09.444847 (+ 947us) server_negotiation.cc:407] Connection header received
0808 03:58:09.446824 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:09.446832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:09.446942 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:09.447047 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:09.448798 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:09.449664 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:09.451598 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:09.452197 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:09.452347 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:09.456242 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:09.456247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:09.456251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:09.456289 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:09.458931 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:09.459731 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:09.459735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:09.459737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:09.459894 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:09.460292 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:09.460295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:09.460297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:09.460580 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:09.460704 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:09.461397 (+ 693us) server_negotiation.cc:299] Negotiation successful
0808 03:58:09.461582 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":258,"threads_started":1}
I20260808 03:58:09.462879 8199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:09.443911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.195:50997)
0808 03:58:09.444665 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:09.444707 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:09.446504 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:09.447333 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:09.447342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:09.447974 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:09.448601 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:09.448621 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:09.449840 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:09.449845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:09.451415 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:09.451430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:09.452458 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:09.452467 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:09.452762 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:09.453577 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:09.453612 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:09.456007 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:09.459076 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:09.459085 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:09.459101 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:09.459625 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:09.459979 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:09.459982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:09.459984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:09.460198 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:09.460675 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:09.460681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:09.461062 (+ 381us) client_negotiation.cc:769] Sending connection context
0808 03:58:09.461298 (+ 236us) client_negotiation.cc:240] Negotiation successful
0808 03:58:09.461721 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":274,"threads_started":1}
I20260808 03:58:09.464435 8193 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40631
I20260808 03:58:09.464751 8193 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:09.465569 8193 heartbeater.cc:511] Master 127.28.251.254:40631 requested a full tablet report, sending...
I20260808 03:58:09.467069 7436 ts_manager.cc:194] Registered new tserver with Master: d37b8ac688c142a6b3b478912340a1e2 (127.28.251.195:34591)
I20260808 03:58:09.467938 7436 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:50997
I20260808 03:58:09.478848 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.251.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.253.keytab.
Entry for principal oryx/127.28.251.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.253.keytab.
I20260808 03:58:09.541313 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.253:35525
--webserver_interface=127.28.251.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.253.keytab
--principal=oryx/127.28.251.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:40631,127.28.251.253:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:09.911360 8202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:09.911753 8202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:09.911926 8202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 03:58:09.917169 7973 heartbeater.cc:503] Master 127.28.251.254:40631 was elected leader, sending a full tablet report...
W20260808 03:58:09.920051 8202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:58:09.920161 8202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:09.920218 8202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:09.920275 8202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:58:09.920318 8202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:58:09.930761 8202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:40631,127.28.251.253:35525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.253.keytab
--principal=oryx/127.28.251.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.253:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.251.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
W20260808 03:58:09.931123 8202 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.251.254:40631,127.28.251.253:35525'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 03:58:09.932343 8202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:09.933727 8202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:09.946049 8210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:09.945982 8214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:09.946025 8211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:09.947232 8202 server_base.cc:1034] running on GCE node
I20260808 03:58:09.948158 8202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:09.949753 8202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:09.951001 8202 hybrid_clock.cc:648] HybridClock initialized: now 1786161489950925 us; error 46 us; skew 500 ppm
I20260808 03:58:09.951380 8202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:09 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161489, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:09.954744 8202 init.cc:377] Logged in from keytab as oryx/127.28.251.253@KRBTEST.COM (short username oryx)
I20260808 03:58:09.956460 8202 webserver.cc:460] Webserver started at http://127.28.251.253:39843/ using document root <none> and password file <none>
I20260808 03:58:09.957213 8202 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:09.957324 8202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:09.957679 8202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:09.960956 8202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data/instance:
uuid: "548398cfe4c049bb8bcbc1e169424ca8"
format_stamp: "Formatted at 2026-08-08 03:58:09 on dist-test-slave-pww0"
server_key: "7ff8880238a47e8651370b336d388d6d"
server_key_iv: "9c2c0fc9b133deab6b266b6324b0bfa8"
server_key_version: "encryptionkey@0"
I20260808 03:58:09.961802 8202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/wal/instance:
uuid: "548398cfe4c049bb8bcbc1e169424ca8"
format_stamp: "Formatted at 2026-08-08 03:58:09 on dist-test-slave-pww0"
server_key: "7ff8880238a47e8651370b336d388d6d"
server_key_iv: "9c2c0fc9b133deab6b266b6324b0bfa8"
server_key_version: "encryptionkey@0"
I20260808 03:58:09.967388 8202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 03:58:09.971515 8227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:09.973038 8202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:58:09.973233 8202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/wal
uuid: "548398cfe4c049bb8bcbc1e169424ca8"
format_stamp: "Formatted at 2026-08-08 03:58:09 on dist-test-slave-pww0"
server_key: "7ff8880238a47e8651370b336d388d6d"
server_key_iv: "9c2c0fc9b133deab6b266b6324b0bfa8"
server_key_version: "encryptionkey@0"
I20260808 03:58:09.973454 8202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:09.990605 8202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:09.995182 8202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:09.995491 8202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:10.014801 8202 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.253:35525
I20260808 03:58:10.014827 8303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.253:35525 every 8 connection(s)
I20260808 03:58:10.016331 8202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
I20260808 03:58:10.020159 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 8202
I20260808 03:58:10.020701 8304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:10.031129 8304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.251.253@KRBTEST.COM
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161489, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.253@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:10.047134 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.022683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.28.251.253:35525)
0808 03:58:10.023533 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.023548 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.023594 (+ 46us) server_negotiation.cc:407] Connection header received
0808 03:58:10.025680 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.025704 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.026144 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.026504 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.028414 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.029798 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.031967 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.032739 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.032896 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.037942 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.037953 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.037968 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.038018 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.041986 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.042792 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.042802 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.042813 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.043014 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.043538 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.043543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.043546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.044256 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.044594 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.045115 (+ 521us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.045615 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":369,"threads_started":1}
I20260808 03:58:10.051250 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.034277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:50084)
0808 03:58:10.035320 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.035372 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.035607 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.036025 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.036033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.036717 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.037068 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.037082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.038352 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.038357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.039727 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.039742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.040541 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.040549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.040760 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.041433 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.041476 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.044119 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.048059 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.048070 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.048077 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.048707 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.049210 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.049213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.049216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.049365 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.050100 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.050105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.050302 (+ 197us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.050623 (+ 321us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.050929 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":581,"threads_started":1}
I20260808 03:58:10.051285 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.034299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.28.251.254:40631)
0808 03:58:10.034853 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.034860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.035505 (+ 645us) server_negotiation.cc:407] Connection header received
0808 03:58:10.035730 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.035733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.035821 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.035916 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.037237 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.038166 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.039884 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.040296 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.040401 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.044356 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.044364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.044369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.044434 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.047919 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.048871 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.048878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.048881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.049065 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.049498 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.049504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.049507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.049964 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.050116 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.050799 (+ 683us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.051046 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":271,"threads_started":1}
I20260808 03:58:10.053050 8304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161489, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.253@KRBTEST.COM for oryx/127.28.251.253@KRBTEST.COM
I20260808 03:58:10.066248 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.053952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:35525 (local address 127.0.0.1:50402)
0808 03:58:10.054119 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.054134 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.054253 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.054525 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.054529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.054923 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.055079 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.055088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.056108 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.056113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.056992 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.057004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.057699 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.057704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.057875 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.058449 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.058467 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.060699 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.063779 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.063784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.063789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.064188 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.064607 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.064612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.064616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.064770 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.065290 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.065294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.065427 (+ 133us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.065559 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.065878 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 03:58:10.066448 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.053882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.28.251.253:35525)
0808 03:58:10.054068 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.054075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.054197 (+ 122us) server_negotiation.cc:407] Connection header received
0808 03:58:10.054360 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.054363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.054419 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.054493 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.055215 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.055966 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.057131 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.057537 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.057653 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.060905 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.060910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.060913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.060947 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.063613 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.064314 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.064321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.064324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.064493 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.064882 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.064885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.064887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.065173 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.065310 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.065665 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.066064 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":58}
I20260808 03:58:10.071653 8304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: Bootstrap starting.
I20260808 03:58:10.075178 8304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:10.076349 8304 log.cc:824] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:10.079579 8304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: No bootstrap required, opened a new log
I20260808 03:58:10.087224 8304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } }
I20260808 03:58:10.087656 8304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:10.087781 8304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548398cfe4c049bb8bcbc1e169424ca8, State: Initialized, Role: FOLLOWER
I20260808 03:58:10.088423 8304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } }
I20260808 03:58:10.089351 8314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } } }
I20260808 03:58:10.089653 8314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 03:58:10.090385 8304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 03:58:10.099931 8330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 03:58:10.100104 8330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 03:58:10.102890 8202 master_runner.cc:428] Detected that this master 548398cfe4c049bb8bcbc1e169424ca8 is joining an existing cluster
I20260808 03:58:10.103021 8202 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.251.253:35525
I20260808 03:58:10.133060 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.108771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.28.251.254:40631)
0808 03:58:10.109093 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.109099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.117018 (+ 7919us) server_negotiation.cc:407] Connection header received
0808 03:58:10.117423 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.117430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.117533 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.117611 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.119758 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.120885 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.122907 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.123640 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.123756 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.126133 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.126140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.126144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.126184 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.129775 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.130745 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.130751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.130754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.130927 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.131353 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.131359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.131362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.131760 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.131876 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.132448 (+ 572us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.132693 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260808 03:58:10.133083 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.108582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.28.251.253:35525)
0808 03:58:10.108959 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.108965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.117457 (+ 8492us) server_negotiation.cc:407] Connection header received
0808 03:58:10.117640 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.117645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.117714 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.117784 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.119707 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.120459 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.122304 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.123031 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.123116 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.126112 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.126119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.126124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.126157 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.129751 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.130707 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.130713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.130715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.130860 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.131299 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.131303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.131306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.131747 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.131876 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.132448 (+ 572us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.132686 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260808 03:58:10.134168 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.108240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:35525 (local address 127.0.0.1:50414)
0808 03:58:10.117373 (+ 9133us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.117392 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.117536 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.117963 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.117967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.118881 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.119538 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.119554 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.120632 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.120635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.122116 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.122132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.123477 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.123485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.123664 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.124426 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.124495 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.125906 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.129908 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.129913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.129917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.130595 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.130987 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.130992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.130995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.131178 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.131944 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.131948 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.132134 (+ 186us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.133370 (+ 1236us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.133682 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8967,"mutex_wait_us":77,"spinlock_wait_cycles":39680,"thread_start_us":8726,"threads_started":1}
I20260808 03:58:10.134559 8345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.108043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:50094)
0808 03:58:10.116833 (+ 8790us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.116861 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.117208 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.117869 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.117873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.118881 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.119539 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.119554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.121051 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.121057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.122717 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.122731 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.124057 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.124061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.124192 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.124712 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.124738 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.125906 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.129908 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.129913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.129917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.130581 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.131069 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.131073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.131076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.131230 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.131961 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.131965 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.132166 (+ 201us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.133736 (+ 1570us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.134050 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8592,"mutex_wait_us":61,"spinlock_wait_cycles":62464,"thread_start_us":8393,"threads_started":1}
WARNING: no policy specified for oryx/127.28.251.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.252@KRBTEST.COM" created.
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161487, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.254@KRBTEST.COM for oryx/127.28.251.253@KRBTEST.COM
I20260808 03:58:10.157428 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.141730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.28.251.253:35525)
0808 03:58:10.141916 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.141923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.142557 (+ 634us) server_negotiation.cc:407] Connection header received
0808 03:58:10.142958 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.142964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.143085 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.143180 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.144469 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.145269 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.146855 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.147351 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.147506 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.151084 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.151090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.151093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.151135 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.154036 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.154844 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.154850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.154852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.155000 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.155500 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.155505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.155508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.155964 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.156126 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.156811 (+ 685us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.157065 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 03:58:10.157521 8351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.141685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:35525 (local address 127.0.0.1:50424)
0808 03:58:10.142408 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.142446 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.142700 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.143255 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.143262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.144041 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.144295 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.144306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.145459 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.145465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.146689 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.146705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.147587 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.147593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.147789 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.148483 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.148504 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.150849 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.154199 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.154218 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.154228 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.154696 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.155120 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.155125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.155129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.155314 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.156133 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.156144 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.156410 (+ 266us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.156661 (+ 251us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.157061 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":266,"threads_started":1}
I20260808 03:58:10.160004 7436 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.251.253:35525: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8"
member_type: NON_VOTER
last_known_addr {
host: "127.28.251.253"
port: 35525
}
attrs {
promote: true
}
}
dest_uuid: "7b56826f1bab45dfb469a8b01ca93516"
cas_config_opid_index: -1
I20260808 03:58:10.161268 7436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } }
I20260808 03:58:10.175411 8354 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 548398cfe4c049bb8bcbc1e169424ca8 (127.28.251.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } } }
I20260808 03:58:10.177438 8271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:10.180416 8354 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.251.253:35525
I20260808 03:58:10.182442 8358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } } }
I20260808 03:58:10.182842 8358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:10.182858 8202 master.cc:537] Master@127.28.251.253:35525 shutting down...
I20260808 03:58:10.183019 8271 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 FOLLOWER]: Refusing update from remote peer 7b56826f1bab45dfb469a8b01ca93516: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 03:58:10.184950 8358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.252.keytab.
Entry for principal oryx/127.28.251.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.252.keytab.
W20260808 03:58:10.191334 7406 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260808 03:58:10.191406 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.252:42623
--webserver_interface=127.28.251.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.252.keytab
--principal=oryx/127.28.251.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:40631,127.28.251.253:35525,127.28.251.252:42623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 03:58:10.200848 8202 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260808 03:58:10.201280 7406 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 -> Peer 548398cfe4c049bb8bcbc1e169424ca8 (127.28.251.253:35525): Couldn't send request to peer 548398cfe4c049bb8bcbc1e169424ca8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 03:58:10.201514 8202 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:10.201818 8202 tablet_replica.cc:330] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: stopping tablet replica
I20260808 03:58:10.208920 8202 master.cc:552] Master@127.28.251.253:35525 shutdown complete.
I20260808 03:58:10.209072 8202 master_runner.cc:305] Clearing existing system tablet
I20260808 03:58:10.209553 8202 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:10.209664 8202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:10.212898 8202 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 03:58:10.214673 8363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:10.215076 8202 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:10.215432 8202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 03:58:10.215577 8202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/wal
uuid: "548398cfe4c049bb8bcbc1e169424ca8"
format_stamp: "Formatted at 2026-08-08 03:58:09 on dist-test-slave-pww0"
server_key: "7ff8880238a47e8651370b336d388d6d"
server_key_iv: "9c2c0fc9b133deab6b266b6324b0bfa8"
server_key_version: "encryptionkey@0"
I20260808 03:58:10.220566 8202 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:10.228260 8202 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 03:58:10.228575 8202 log.cc:1196] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 03:58:10.229221 8202 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: Deleting consensus metadata
I20260808 03:58:10.229595 8202 master_runner.cc:315] Copying system tablet from 127.28.251.254:40631
I20260808 03:58:10.232586 8202 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: tablet copy: Beginning tablet copy session from remote peer at address 127.28.251.254:40631
I20260808 03:58:10.255496 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.233539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.28.251.254:40631)
0808 03:58:10.233727 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.233733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.240268 (+ 6535us) server_negotiation.cc:407] Connection header received
0808 03:58:10.240483 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.240489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.240586 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.240712 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.241833 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.242989 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.245021 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.245711 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.245853 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.248319 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.248327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.248332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.248380 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.252047 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.253045 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.253051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.253055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.253239 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.253691 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.253696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.253700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.254100 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.254272 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.254825 (+ 553us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.255074 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 03:58:10.255546 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.233491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:50106)
0808 03:58:10.240091 (+ 6600us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.240115 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.240326 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.240771 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.240776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.241334 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.241653 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.241666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.243195 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.243201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.244756 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.244807 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.245896 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.245902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.246102 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.246850 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.246875 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.248075 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.252265 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.252271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.252275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.252881 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.253369 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.253374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.253377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.253566 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.254251 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.254255 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.254469 (+ 214us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.254673 (+ 204us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.255042 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6423,"thread_start_us":6239,"threads_started":1}
I20260808 03:58:10.256815 7474 tablet_copy_service.cc:140] P 7b56826f1bab45dfb469a8b01ca93516: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 548398cfe4c049bb8bcbc1e169424ca8 ({username='oryx', principal='oryx/127.28.251.253@KRBTEST.COM'} at 127.0.0.1:50106)
I20260808 03:58:10.257144 7474 tablet_copy_service.cc:161] P 7b56826f1bab45dfb469a8b01ca93516: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 548398cfe4c049bb8bcbc1e169424ca8 at {username='oryx', principal='oryx/127.28.251.253@KRBTEST.COM'} at 127.0.0.1:50106: session id = 548398cfe4c049bb8bcbc1e169424ca8-00000000000000000000000000000000
I20260808 03:58:10.261857 7474 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 03:58:10.264674 8202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 03:58:10.272197 8202 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: tablet copy: Starting download of 0 data blocks...
I20260808 03:58:10.272576 8202 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: tablet copy: Starting download of 1 WAL segments...
I20260808 03:58:10.276082 8202 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 03:58:10.281117 7474 tablet_copy_service.cc:342] P 7b56826f1bab45dfb469a8b01ca93516: Request end of tablet copy session 548398cfe4c049bb8bcbc1e169424ca8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.251.253@KRBTEST.COM'} at 127.0.0.1:50106
I20260808 03:58:10.281384 7474 tablet_copy_service.cc:433] P 7b56826f1bab45dfb469a8b01ca93516: ending tablet copy session 548398cfe4c049bb8bcbc1e169424ca8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 548398cfe4c049bb8bcbc1e169424ca8
I20260808 03:58:10.294727 8202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:10.297757 8373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:10.308408 8375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:10.308719 8377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:10.308920 8202 server_base.cc:1034] running on GCE node
I20260808 03:58:10.309522 8202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:10.310209 8202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:10.311429 8202 hybrid_clock.cc:648] HybridClock initialized: now 1786161490311361 us; error 37 us; skew 500 ppm
I20260808 03:58:10.311749 8202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161490, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:10.313663 8202 init.cc:377] Logged in from keytab as oryx/127.28.251.253@KRBTEST.COM (short username oryx)
I20260808 03:58:10.315042 8202 webserver.cc:460] Webserver started at http://127.28.251.253:39649/ using document root <none> and password file <none>
I20260808 03:58:10.315374 8202 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:10.315450 8202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:10.317956 8202 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 03:58:10.319628 8390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:10.320047 8202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 03:58:10.320189 8202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/wal
uuid: "548398cfe4c049bb8bcbc1e169424ca8"
format_stamp: "Formatted at 2026-08-08 03:58:09 on dist-test-slave-pww0"
server_key: "7ff8880238a47e8651370b336d388d6d"
server_key_iv: "9c2c0fc9b133deab6b266b6324b0bfa8"
server_key_version: "encryptionkey@0"
I20260808 03:58:10.320356 8202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:10.339185 8202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:10.342718 8202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:10.374434 8202 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.253:35525
I20260808 03:58:10.374473 8476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.253:35525 every 8 connection(s)
I20260808 03:58:10.375780 8202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
I20260808 03:58:10.381911 8477 sys_catalog.cc:263] Verifying existing consensus state
I20260808 03:58:10.386181 8477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: Bootstrap starting.
I20260808 03:58:10.405654 8477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 03:58:10.406234 8477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: Bootstrap complete.
I20260808 03:58:10.407513 8477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } }
I20260808 03:58:10.407816 8477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 548398cfe4c049bb8bcbc1e169424ca8, State: Initialized, Role: LEARNER
I20260808 03:58:10.408145 8477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } }
I20260808 03:58:10.409415 8477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:58:10.409170 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } } }
I20260808 03:58:10.409749 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: This master's current role is: LEARNER
I20260808 03:58:10.419095 8499 catalog_manager.cc:1259] Loaded cluster ID: 66b7c2628b5a49eb9ab7cf57e481cb60
I20260808 03:58:10.419260 8499 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: loading cluster ID for follower catalog manager: success
I20260808 03:58:10.423247 8499 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: acquiring CA information for follower catalog manager: success
I20260808 03:58:10.426496 8499 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 03:58:10.470706 8193 heartbeater.cc:503] Master 127.28.251.254:40631 was elected leader, sending a full tablet report...
W20260808 03:58:10.626145 8361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:10.626539 8361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:10.626688 8361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:10.634516 8361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:58:10.634651 8361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:10.634729 8361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:10.634810 8361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:58:10.634868 8361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:58:10.645929 8361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:40631,127.28.251.253:35525,127.28.251.252:42623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.252.keytab
--principal=oryx/127.28.251.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.252:42623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.251.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:10.647399 8361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:10.649240 8361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 03:58:10.662125 8351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.653537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:35525 (local address 127.0.0.1:50432)
0808 03:58:10.653806 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.653835 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.653956 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.654891 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.654896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.654941 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:10.655157 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.655171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.657162 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.657169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.659299 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.659321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.661221 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.661229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.661493 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.661598 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.661676 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.661842 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260808 03:58:10.663396 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.653793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.28.251.253:35525)
0808 03:58:10.654489 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.654495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.654515 (+ 20us) server_negotiation.cc:407] Connection header received
0808 03:58:10.654586 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.654592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.654685 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.654802 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:10.655333 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.656962 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.659612 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.660972 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.662639 (+ 1667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.662853 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.662915 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.663065 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":404,"threads_started":1}
I20260808 03:58:10.664609 8433 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260808 03:58:10.665608 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b56826f1bab45dfb469a8b01ca93516. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } } }
I20260808 03:58:10.665897 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: This master's current role is: LEARNER
W20260808 03:58:10.665796 8507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:10.667205 8508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:10.668772 8510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:10.669621 8361 server_base.cc:1034] running on GCE node
I20260808 03:58:10.670585 8361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:10.672273 8361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:10.673494 8361 hybrid_clock.cc:648] HybridClock initialized: now 1786161490673439 us; error 45 us; skew 500 ppm
I20260808 03:58:10.673889 8361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161490, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:10.677404 8361 init.cc:377] Logged in from keytab as oryx/127.28.251.252@KRBTEST.COM (short username oryx)
I20260808 03:58:10.679167 8361 webserver.cc:460] Webserver started at http://127.28.251.252:45955/ using document root <none> and password file <none>
I20260808 03:58:10.680081 8361 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:10.680197 8361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:10.680488 8361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:10.683897 8361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data/instance:
uuid: "e4c1cfe150f14368b3815cbaf16832d4"
format_stamp: "Formatted at 2026-08-08 03:58:10 on dist-test-slave-pww0"
server_key: "1d143b0ee3a03d64598a05a1f950fa1c"
server_key_iv: "719ea798f73cc9d0e7ef61378e2778c4"
server_key_version: "encryptionkey@0"
I20260808 03:58:10.684815 8361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/wal/instance:
uuid: "e4c1cfe150f14368b3815cbaf16832d4"
format_stamp: "Formatted at 2026-08-08 03:58:10 on dist-test-slave-pww0"
server_key: "1d143b0ee3a03d64598a05a1f950fa1c"
server_key_iv: "719ea798f73cc9d0e7ef61378e2778c4"
server_key_version: "encryptionkey@0"
I20260808 03:58:10.690729 8361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 03:58:10.694710 8522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:10.696357 8361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:10.696547 8361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/wal
uuid: "e4c1cfe150f14368b3815cbaf16832d4"
format_stamp: "Formatted at 2026-08-08 03:58:10 on dist-test-slave-pww0"
server_key: "1d143b0ee3a03d64598a05a1f950fa1c"
server_key_iv: "719ea798f73cc9d0e7ef61378e2778c4"
server_key_version: "encryptionkey@0"
I20260808 03:58:10.696782 8361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:10.721690 8361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:10.726605 8361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:10.726917 8361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:10.745050 8361 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.252:42623
I20260808 03:58:10.745088 8601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.252:42623 every 8 connection(s)
I20260808 03:58:10.746526 8361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
I20260808 03:58:10.750597 8603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:10.754988 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 8361
I20260808 03:58:10.758882 8603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.251.252@KRBTEST.COM
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161490, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.252@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:10.780764 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.764095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50120 (local address 127.28.251.254:40631)
0808 03:58:10.764687 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.764693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.765285 (+ 592us) server_negotiation.cc:407] Connection header received
0808 03:58:10.765468 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.765472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.765540 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.765633 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.766993 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.767825 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.769379 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.770027 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.770171 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.774551 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.774558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.774564 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.774612 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.777629 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.778404 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.778410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.778413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.778547 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.778961 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.778966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.778970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.779295 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.779437 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.780132 (+ 695us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.780383 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":292,"threads_started":1}
I20260808 03:58:10.781970 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.764104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:50120)
0808 03:58:10.765094 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.765155 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.765359 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.765743 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.765751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.766514 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.766813 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.766823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.767965 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.767971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.769210 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.769225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.770307 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.770317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.770560 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.771290 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.771331 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.774296 (+ 2965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.777772 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.777786 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.777801 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.778278 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.778657 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.778662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.778666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.778847 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.779435 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.779444 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.779674 (+ 230us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.779942 (+ 268us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.780734 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":141,"thread_start_us":517,"threads_started":1}
I20260808 03:58:10.782019 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.757674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.28.251.252:42623)
0808 03:58:10.758577 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.758590 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.758638 (+ 48us) server_negotiation.cc:407] Connection header received
0808 03:58:10.760633 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.760663 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.761207 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.761648 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.763677 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.765801 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.767143 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.767884 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.768101 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.773192 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.773207 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.773228 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.773300 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.776936 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.777936 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.777949 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.777952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.778174 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.778657 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.778663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.778665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.779364 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.779723 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.780155 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.780585 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"spinlock_wait_cycles":20736,"thread_start_us":426,"threads_started":1}
I20260808 03:58:10.783586 8603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161490, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.252@KRBTEST.COM for oryx/127.28.251.253@KRBTEST.COM
I20260808 03:58:10.797667 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.784550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.28.251.253:35525)
0808 03:58:10.784779 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.784786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.784811 (+ 25us) server_negotiation.cc:407] Connection header received
0808 03:58:10.784883 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.784888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.784970 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.785071 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.785968 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.786783 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.788193 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.788673 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.788813 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.792230 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.792236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.792239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.792278 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.794885 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.795668 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.795673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.795676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.795820 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.796229 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.796232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.796234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.796528 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.796678 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.797053 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.797311 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 03:58:10.797667 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.784374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:35525 (local address 127.0.0.1:50436)
0808 03:58:10.784574 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.784593 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.784696 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.785207 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.785213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.785622 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.785785 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.785796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.786945 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.786949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.788012 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.788027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.788843 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.788849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.789080 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.789739 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.789758 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.791997 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.795054 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.795058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.795061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.795530 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.795941 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.795946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.795950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.796101 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.796649 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.796654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.796807 (+ 153us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.796921 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.797270 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 03:58:10.798866 8603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161490, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.252@KRBTEST.COM for oryx/127.28.251.252@KRBTEST.COM
I20260808 03:58:10.812482 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.799663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:42623 (local address 127.0.0.1:46816)
0808 03:58:10.799861 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.799883 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.800018 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.800341 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.800346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.800780 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.800997 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.801008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.802087 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.802107 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.802984 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.802997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.803829 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.803833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.803988 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.804571 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.804590 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.806822 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.809846 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.809850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.809853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.810280 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.810688 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.810692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.810694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.810925 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.811509 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.811513 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.811676 (+ 163us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.811817 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.812075 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 03:58:10.812498 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.799713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.28.251.252:42623)
0808 03:58:10.799861 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.799867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.799948 (+ 81us) server_negotiation.cc:407] Connection header received
0808 03:58:10.800148 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.800153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.800218 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.800306 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.801152 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.801960 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.803142 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.803685 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.803804 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.807050 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.807055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.807059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.807092 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.809677 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.810393 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.810398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.810400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.810535 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.811046 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.811051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.811055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.811361 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.811514 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.811939 (+ 425us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.812200 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":51}
I20260808 03:58:10.817646 8603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: Bootstrap starting.
I20260808 03:58:10.821147 8603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:10.822263 8603 log.cc:824] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:10.825644 8603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: No bootstrap required, opened a new log
I20260808 03:58:10.833455 8603 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } }
I20260808 03:58:10.833846 8603 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:10.833973 8603 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c1cfe150f14368b3815cbaf16832d4, State: Initialized, Role: FOLLOWER
I20260808 03:58:10.834641 8603 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } }
I20260808 03:58:10.835741 8613 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } } }
I20260808 03:58:10.836225 8613 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 03:58:10.837090 8603 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 03:58:10.846757 8630 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 03:58:10.846944 8630 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 03:58:10.852106 8361 master_runner.cc:428] Detected that this master e4c1cfe150f14368b3815cbaf16832d4 is joining an existing cluster
I20260808 03:58:10.852249 8361 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.251.252:42623
I20260808 03:58:10.874987 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.856694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.28.251.254:40631)
0808 03:58:10.856909 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.856916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.857784 (+ 868us) server_negotiation.cc:407] Connection header received
0808 03:58:10.858116 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.858123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.858216 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.858296 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.860037 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.861212 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.863438 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.864140 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.864266 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.867619 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.867626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.867631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.867678 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.871392 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.872345 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.872352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.872356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.872528 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.873076 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.873082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.873085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.873450 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.873578 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.874347 (+ 769us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.874613 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 03:58:10.876331 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.856771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:50132)
0808 03:58:10.857631 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.857653 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.857934 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.858544 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.858548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.859359 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.859812 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.859832 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.861460 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.861466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.863231 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.863251 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.864605 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.864613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.864846 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.865646 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.865671 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.867311 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.871619 (+ 4308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.871625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.871629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.872191 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.872706 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.872724 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.872728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.872939 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.873688 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.873694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.874004 (+ 310us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.875300 (+ 1296us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.875769 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":499,"threads_started":1}
I20260808 03:58:10.883318 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.858108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.28.251.252:42623)
0808 03:58:10.858300 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.858306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.865613 (+ 7307us) server_negotiation.cc:407] Connection header received
0808 03:58:10.865821 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.865825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.865900 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.865968 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.867918 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.869100 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.871555 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.872283 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.872364 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.875555 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.875565 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.875570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.875612 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.879757 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.880892 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.880897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.880901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.881091 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.881500 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.881505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.881508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.881978 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.882113 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.882688 (+ 575us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.882953 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"spinlock_wait_cycles":1792}
I20260808 03:58:10.883915 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.856714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.28.251.253:35525)
0808 03:58:10.856910 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.856915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.864727 (+ 7812us) server_negotiation.cc:407] Connection header received
0808 03:58:10.865661 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.865668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.865758 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.865829 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.867939 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.869100 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.871566 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.872211 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.872319 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.875559 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.875565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.875571 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.875612 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.880732 (+ 5120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.881533 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.881539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.881542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.881700 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.882160 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.882166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.882169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.882567 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.882700 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.883269 (+ 569us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.883571 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 03:58:10.885042 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.857297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:35525 (local address 127.0.0.1:50440)
0808 03:58:10.864563 (+ 7266us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.864591 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.865459 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.866066 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.866070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.867186 (+ 1116us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.867748 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.867763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.869286 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.869292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.871348 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.871366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.872755 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.872763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.872979 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.873785 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.873811 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.875268 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.880895 (+ 5627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.880900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.880903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.881393 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.881855 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.881859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.881862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.882025 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.882776 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.882781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.882973 (+ 192us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.884192 (+ 1219us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.884574 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7065,"mutex_wait_us":192,"spinlock_wait_cycles":655360,"thread_start_us":7439,"threads_started":1}
I20260808 03:58:10.892057 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.858471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:42623 (local address 127.0.0.1:46818)
0808 03:58:10.865521 (+ 7050us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.865549 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.865695 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.866153 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.866162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.867186 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.867749 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.867763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.869306 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.869310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.871348 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.871366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.872738 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.872749 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.872964 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.873699 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.873750 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.875371 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.879937 (+ 4566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.879942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.879946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.880765 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.881230 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.881235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.881238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.881381 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.882176 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.882181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.882345 (+ 164us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.891075 (+ 8730us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.891513 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6904,"mutex_wait_us":222,"spinlock_wait_cycles":42112,"thread_start_us":6738,"threads_started":1}
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161487, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.254@KRBTEST.COM for oryx/127.28.251.252@KRBTEST.COM
I20260808 03:58:10.897338 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 7533
I20260808 03:58:10.902118 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.883554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.28.251.252:42623)
0808 03:58:10.883969 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.884204 (+ 235us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.884223 (+ 19us) server_negotiation.cc:407] Connection header received
0808 03:58:10.884324 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.884329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.884467 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.885915 (+ 1448us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.886142 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.887526 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.889390 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.890144 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.890899 (+ 755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.894390 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.894398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.894402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.894444 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.898368 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.899254 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.899258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.899261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.899385 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.899901 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.899906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.899909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.900346 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.900503 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.901415 (+ 912us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.901705 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":12134,"threads_started":1}
I20260808 03:58:10.902930 8351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.882806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:42623 (local address 127.0.0.1:46826)
0808 03:58:10.883196 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.883215 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.883323 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.884672 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.884678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:10.885187 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:10.885394 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.885405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.887699 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.887705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.889080 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:10.889100 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.890378 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.890385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:10.890564 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.891242 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:10.891266 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:10.894162 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:10.898572 (+ 4410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.898579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.898583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.899106 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.899551 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:10.899557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:10.899561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:10.899749 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:10.900516 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:10.900522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:10.900742 (+ 220us) client_negotiation.cc:769] Sending connection context
0808 03:58:10.900930 (+ 188us) client_negotiation.cc:240] Negotiation successful
0808 03:58:10.902515 (+ 1585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260808 03:58:10.906087 7436 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.251.252:42623: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4"
member_type: NON_VOTER
last_known_addr {
host: "127.28.251.252"
port: 42623
}
attrs {
promote: true
}
}
dest_uuid: "7b56826f1bab45dfb469a8b01ca93516"
cas_config_opid_index: 5
I20260808 03:58:10.907334 7436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } }
I20260808 03:58:10.909430 8358 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4c1cfe150f14368b3815cbaf16832d4 (127.28.251.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
Aug 08 03:58:10 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.251.193@KRBTEST.COM
I20260808 03:58:10.912395 8433 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 LEARNER]: Refusing update from remote peer 7b56826f1bab45dfb469a8b01ca93516: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 03:58:10.915405 8358 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.251.252:42623
I20260808 03:58:10.916119 8361 master.cc:537] Master@127.28.251.252:42623 shutting down...
I20260808 03:58:10.917250 8564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:10.921411 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.898527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.28.251.193:43373)
0808 03:58:10.900046 (+ 1519us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.900055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.900089 (+ 34us) server_negotiation.cc:407] Connection header received
0808 03:58:10.900163 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.900172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.900499 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.900666 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:10.902186 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.903522 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.904895 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:10.905777 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:10.905904 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:10.911946 (+ 6042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:10.911970 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:10.911995 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:10.912062 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:10.916476 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.917540 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.917551 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.917555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.917756 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:10.918319 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:10.918324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:10.918327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:10.918790 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:10.920396 (+ 1606us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:10.920579 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 03:58:10.920969 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1311,"thread_start_us":1119,"threads_started":1}
I20260808 03:58:10.921861 8564 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 FOLLOWER]: Refusing update from remote peer 7b56826f1bab45dfb469a8b01ca93516: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 03:58:10.923044 7738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:10.927251 8647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 03:58:10.936286 8358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:10.938889 8653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
I20260808 03:58:10.939513 8653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:10.941514 8433 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4c1cfe150f14368b3815cbaf16832d4 (127.28.251.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
I20260808 03:58:10.946466 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: NON_VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: true } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
I20260808 03:58:10.946787 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: This master's current role is: LEARNER
I20260808 03:58:10.947887 8647 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516: attempting to promote NON_VOTER 548398cfe4c049bb8bcbc1e169424ca8 to VOTER
I20260808 03:58:10.948772 8647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } }
I20260808 03:58:10.951268 8433 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 LEARNER]: Refusing update from remote peer 7b56826f1bab45dfb469a8b01ca93516: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 03:58:10.951928 8647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 03:58:10.954741 8653 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 548398cfe4c049bb8bcbc1e169424ca8 (127.28.251.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
I20260808 03:58:10.959529 8433 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 548398cfe4c049bb8bcbc1e169424ca8 (127.28.251.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
I20260808 03:58:10.960614 8647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
I20260808 03:58:10.960568 8648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
I20260808 03:58:10.961037 8648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:10.961073 8647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:10.966221 8361 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:10.966745 8361 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:10.967013 8361 tablet_replica.cc:330] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: stopping tablet replica
I20260808 03:58:10.966885 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
I20260808 03:58:10.967160 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 03:58:10.970497 7403 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 -> Peer e4c1cfe150f14368b3815cbaf16832d4 (127.28.251.252:42623): Couldn't send request to peer e4c1cfe150f14368b3815cbaf16832d4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 03:58:10.974366 8361 master.cc:552] Master@127.28.251.252:42623 shutdown complete.
I20260808 03:58:10.974453 8361 master_runner.cc:305] Clearing existing system tablet
I20260808 03:58:10.974802 8361 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:10.974910 8361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:10.977331 8361 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260808 03:58:10.978825 8660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:10.979185 8361 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:10.979496 8361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 03:58:10.979604 8361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/wal
uuid: "e4c1cfe150f14368b3815cbaf16832d4"
format_stamp: "Formatted at 2026-08-08 03:58:10 on dist-test-slave-pww0"
server_key: "1d143b0ee3a03d64598a05a1f950fa1c"
server_key_iv: "719ea798f73cc9d0e7ef61378e2778c4"
server_key_version: "encryptionkey@0"
I20260808 03:58:10.983327 8361 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:10.988996 8361 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 03:58:10.989192 8361 log.cc:1196] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 03:58:10.989645 8361 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: Deleting consensus metadata
I20260808 03:58:10.989902 8361 master_runner.cc:315] Copying system tablet from 127.28.251.254:40631
I20260808 03:58:10.997231 8361 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: tablet copy: Beginning tablet copy session from remote peer at address 127.28.251.254:40631
I20260808 03:58:11.011544 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.998200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.28.251.254:40631)
0808 03:58:10.998416 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:10.998424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:10.998981 (+ 557us) server_negotiation.cc:407] Connection header received
0808 03:58:10.999185 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:10.999191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:10.999269 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:10.999368 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:11.000595 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.001628 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.002966 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.003427 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.003589 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:11.005515 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:11.005521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:11.005524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:11.005561 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:11.008402 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:11.009131 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:11.009136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:11.009138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:11.009284 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:11.009778 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:11.009783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:11.009787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:11.010110 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:11.010253 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:11.010781 (+ 528us) server_negotiation.cc:299] Negotiation successful
0808 03:58:11.011022 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 03:58:11.011546 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:10.998161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:50140)
0808 03:58:10.998849 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:10.998868 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:10.999037 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:10.999438 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:10.999443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:11.000089 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:11.000425 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:11.000439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.001806 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.001812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:11.002816 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:11.002830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.003615 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.003621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:11.003822 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:11.004477 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:11.004496 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:11.005300 (+ 804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:11.008568 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:11.008574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:11.008579 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:11.009003 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:11.009404 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:11.009409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:11.009413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:11.009664 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:11.010234 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:11.010239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:11.010448 (+ 209us) client_negotiation.cc:769] Sending connection context
0808 03:58:11.010576 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 03:58:11.011110 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":381,"threads_started":1}
I20260808 03:58:11.012674 7474 tablet_copy_service.cc:140] P 7b56826f1bab45dfb469a8b01ca93516: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e4c1cfe150f14368b3815cbaf16832d4 ({username='oryx', principal='oryx/127.28.251.252@KRBTEST.COM'} at 127.0.0.1:50140)
I20260808 03:58:11.012849 7474 tablet_copy_service.cc:161] P 7b56826f1bab45dfb469a8b01ca93516: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e4c1cfe150f14368b3815cbaf16832d4 at {username='oryx', principal='oryx/127.28.251.252@KRBTEST.COM'} at 127.0.0.1:50140: session id = e4c1cfe150f14368b3815cbaf16832d4-00000000000000000000000000000000
I20260808 03:58:11.015316 7474 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 03:58:11.017333 8361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 03:58:11.023269 8361 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: tablet copy: Starting download of 0 data blocks...
I20260808 03:58:11.023620 8361 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: tablet copy: Starting download of 1 WAL segments...
I20260808 03:58:11.026091 8361 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 03:58:11.029758 7474 tablet_copy_service.cc:342] P 7b56826f1bab45dfb469a8b01ca93516: Request end of tablet copy session e4c1cfe150f14368b3815cbaf16832d4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.251.252@KRBTEST.COM'} at 127.0.0.1:50140
I20260808 03:58:11.029901 7474 tablet_copy_service.cc:433] P 7b56826f1bab45dfb469a8b01ca93516: ending tablet copy session e4c1cfe150f14368b3815cbaf16832d4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e4c1cfe150f14368b3815cbaf16832d4
I20260808 03:58:11.040915 8361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:11.043534 8673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:11.048985 8674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:11.056175 8676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:11.057358 8361 server_base.cc:1034] running on GCE node
I20260808 03:58:11.057723 8361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:11.058679 8361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:11.059993 8361 hybrid_clock.cc:648] HybridClock initialized: now 1786161491059949 us; error 31 us; skew 500 ppm
I20260808 03:58:11.060254 8361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:11 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161491, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:11.062122 8361 init.cc:377] Logged in from keytab as oryx/127.28.251.252@KRBTEST.COM (short username oryx)
I20260808 03:58:11.063550 8361 webserver.cc:460] Webserver started at http://127.28.251.252:43585/ using document root <none> and password file <none>
I20260808 03:58:11.063877 8361 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:11.063977 8361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:11.066447 8361 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 03:58:11.068814 8689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:11.069271 8361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 03:58:11.069411 8361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/wal
uuid: "e4c1cfe150f14368b3815cbaf16832d4"
format_stamp: "Formatted at 2026-08-08 03:58:10 on dist-test-slave-pww0"
server_key: "1d143b0ee3a03d64598a05a1f950fa1c"
server_key_iv: "719ea798f73cc9d0e7ef61378e2778c4"
server_key_version: "encryptionkey@0"
I20260808 03:58:11.069635 8361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:11.069934 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 7533
I20260808 03:58:11.115906 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:43373
--local_ip_for_outbound_sockets=127.28.251.193
--tserver_master_addrs=127.28.251.254:40631
--webserver_port=34225
--webserver_interface=127.28.251.193
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab
--principal=oryx/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.251.254:40631,127.28.251.253:35525
--tserver_master_addrs=127.28.251.254:40631,127.28.251.253:35525,127.28.251.252:42623 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 03:58:11.116850 8361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:11.120483 8361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:11.182443 8361 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.252:42623
I20260808 03:58:11.182498 8768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.252:42623 every 8 connection(s)
I20260808 03:58:11.183506 8361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
I20260808 03:58:11.189504 8769 sys_catalog.cc:263] Verifying existing consensus state
I20260808 03:58:11.194204 8769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: Bootstrap starting.
I20260808 03:58:11.211615 8769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 03:58:11.212165 8769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: Bootstrap complete.
I20260808 03:58:11.213282 8769 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } }
I20260808 03:58:11.213613 8769 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4c1cfe150f14368b3815cbaf16832d4, State: Initialized, Role: LEARNER
I20260808 03:58:11.213909 8769 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } }
I20260808 03:58:11.214910 8779 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
I20260808 03:58:11.215302 8769 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:58:11.215387 8779 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: This master's current role is: LEARNER
I20260808 03:58:11.227411 8800 catalog_manager.cc:1259] Loaded cluster ID: 66b7c2628b5a49eb9ab7cf57e481cb60
I20260808 03:58:11.227598 8800 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: loading cluster ID for follower catalog manager: success
I20260808 03:58:11.230253 8800 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: acquiring CA information for follower catalog manager: success
I20260808 03:58:11.232282 8800 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 03:58:11.457413 8805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:11.448107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.28.251.252:42623)
0808 03:58:11.450622 (+ 2515us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:11.450627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:11.450649 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:58:11.450720 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:11.450724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:11.450806 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:11.450936 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:11.451517 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.452594 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.455031 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.456249 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.456451 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:11.456622 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:11.456965 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 03:58:11.457100 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2392,"thread_start_us":2237,"threads_started":1}
I20260808 03:58:11.457444 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:11.447897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:42623 (local address 127.0.0.1:46840)
0808 03:58:11.450118 (+ 2221us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:11.450143 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:11.450326 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:11.450983 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:11.450988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:11.451020 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:11.451251 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:11.451263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.452784 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.452791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:11.454723 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:11.454755 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.456507 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.456516 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:11.456732 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:11.456782 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 03:58:11.456925 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 03:58:11.457099 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2075,"thread_start_us":1964,"threads_started":1}
I20260808 03:58:11.458681 8736 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 03:58:11.459483 8779 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b56826f1bab45dfb469a8b01ca93516. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
I20260808 03:58:11.459828 8779 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: This master's current role is: LEARNER
W20260808 03:58:11.568353 8699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:11.568687 8699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:11.568817 8699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:11.578938 8699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:11.579118 8699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:11.579316 8699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:58:11.595861 8699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab
--principal=oryx/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:43373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=34225
--tserver_master_addrs=127.28.251.254:40631,127.28.251.253:35525,127.28.251.252:42623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:11.597633 8699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:11.599539 8699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:11.617254 8813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:11.618793 8814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:11.618809 8816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:11.619904 8699 server_base.cc:1034] running on GCE node
I20260808 03:58:11.620730 8699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:11.622007 8699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:11.623265 8699 hybrid_clock.cc:648] HybridClock initialized: now 1786161491623173 us; error 85 us; skew 500 ppm
I20260808 03:58:11.623716 8699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:11 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161491, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:11.628690 8699 init.cc:377] Logged in from keytab as oryx/127.28.251.193@KRBTEST.COM (short username oryx)
I20260808 03:58:11.631011 8699 webserver.cc:460] Webserver started at http://127.28.251.193:34225/ using document root <none> and password file <none>
I20260808 03:58:11.632159 8699 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:11.632364 8699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:11.638677 8699 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 03:58:11.642009 8829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:11.643723 8699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:11.643899 8699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "446ce0b231b841a2875981263d95cad2"
format_stamp: "Formatted at 2026-08-08 03:58:08 on dist-test-slave-pww0"
server_key: "7816e2283dcd4d2837424a7b4041b508"
server_key_iv: "193c59dafc3c41cd822923a24f0f7b0e"
server_key_version: "encryptionkey@0"
I20260808 03:58:11.644994 8699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:11.676069 8699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:11.681077 8699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:11.681387 8699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:11.682487 8699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:11.684087 8699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:11.684182 8699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:11.684288 8699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:11.684378 8699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:11.720449 8699 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:43373
I20260808 03:58:11.720518 9005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:43373 every 8 connection(s)
I20260808 03:58:11.722230 8699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:58:11.729735 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 8699
I20260808 03:58:11.730160 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 7754
Aug 08 03:58:11 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161491, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.253@KRBTEST.COM
Aug 08 03:58:11 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.251.194@KRBTEST.COM
Aug 08 03:58:11 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161491, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.252@KRBTEST.COM
Aug 08 03:58:11 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161491, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:11.755841 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:11.725493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:56875 (local address 127.28.251.253:35525)
0808 03:58:11.726185 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:11.726190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:11.727664 (+ 1474us) server_negotiation.cc:407] Connection header received
0808 03:58:11.732644 (+ 4980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:11.732651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:11.732773 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:11.732870 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:11.734153 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.735272 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.738257 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.739022 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.739129 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:11.746971 (+ 7842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:11.746981 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:11.746986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:11.747043 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:11.750869 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:11.752176 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:11.752183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:11.752187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:11.752387 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:11.753199 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:11.753206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:11.753210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:11.753618 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:11.753762 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:11.755012 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0808 03:58:11.755316 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":445,"threads_started":1}
I20260808 03:58:11.755896 8805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:11.727015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:52375 (local address 127.28.251.252:42623)
0808 03:58:11.727210 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:11.727214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:11.727826 (+ 612us) server_negotiation.cc:407] Connection header received
0808 03:58:11.730609 (+ 2783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:11.730616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:11.730723 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:11.730838 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:11.733891 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.735057 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.738257 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.738992 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.739131 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:11.748358 (+ 9227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:11.748366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:11.748371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:11.748426 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:11.752432 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:11.753422 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:11.753428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:11.753432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:11.753607 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:11.754070 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:11.754076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:11.754080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:11.754472 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:11.754612 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:11.755208 (+ 596us) server_negotiation.cc:299] Negotiation successful
0808 03:58:11.755524 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 03:58:11.758677 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:11.726211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:35525 (local address 127.28.251.193:56875)
0808 03:58:11.727426 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:11.727512 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:11.730120 (+ 2608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:11.733141 (+ 3021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:11.733146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:11.733644 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:11.733952 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:11.733966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.735528 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.735534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:11.738042 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:11.738060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.739622 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.739632 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:11.739997 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:11.741240 (+ 1243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:11.741280 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:11.746612 (+ 5332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:11.751118 (+ 4506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:11.751131 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:11.751153 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:11.751970 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:11.752579 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:11.752584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:11.752588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:11.752995 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:11.753874 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:11.753883 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:11.754441 (+ 558us) client_negotiation.cc:769] Sending connection context
0808 03:58:11.756160 (+ 1719us) client_negotiation.cc:240] Negotiation successful
0808 03:58:11.757117 (+ 957us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":477,"thread_start_us":873,"threads_started":1}
I20260808 03:58:11.758723 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:11.726987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:42623 (local address 127.28.251.193:52375)
0808 03:58:11.727715 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:11.727734 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:11.730121 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:11.731506 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:11.731521 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:11.732611 (+ 1090us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:11.733621 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:11.733652 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.735310 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.735315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:11.738042 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:11.738060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.740541 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.740554 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:11.740750 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:11.743093 (+ 2343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:11.743125 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:11.747768 (+ 4643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:11.752638 (+ 4870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:11.752644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:11.752648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:11.753228 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:11.753773 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:11.753778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:11.753782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:11.753936 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:11.754706 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:11.754712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:11.754882 (+ 170us) client_negotiation.cc:769] Sending connection context
0808 03:58:11.756186 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0808 03:58:11.756915 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":175,"spinlock_wait_cycles":46464,"thread_start_us":1103,"threads_started":1}
I20260808 03:58:11.758834 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:11.731815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.28.251.194:44113)
0808 03:58:11.732573 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:11.732583 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:11.732618 (+ 35us) server_negotiation.cc:407] Connection header received
0808 03:58:11.732693 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:11.732702 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:11.732978 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:11.735392 (+ 2414us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:11.735753 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.737455 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.739630 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.740621 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.740766 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:11.747905 (+ 7139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:11.747933 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:11.747953 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:11.748004 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:11.753536 (+ 5532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:11.754759 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:11.754772 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:11.754775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:11.754986 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:11.755586 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:11.755593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:11.755640 (+ 47us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:11.756149 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:11.756500 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:11.758017 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0808 03:58:11.758339 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":427,"threads_started":1}
I20260808 03:58:11.760504 7966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:11.760926 9014 heartbeater.cc:347] Connected to a master server at 127.28.251.253:35525
I20260808 03:58:11.761300 9014 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:11.762367 9014 heartbeater.cc:511] Master 127.28.251.253:35525 requested a full tablet report, sending...
I20260808 03:58:11.763264 9007 heartbeater.cc:347] Connected to a master server at 127.28.251.252:42623
I20260808 03:58:11.763412 9007 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:11.763919 9007 heartbeater.cc:511] Master 127.28.251.252:42623 requested a full tablet report, sending...
I20260808 03:58:11.764042 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:11.728484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.193:38069)
0808 03:58:11.729125 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:11.729145 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:11.730498 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:11.744325 (+ 13827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:11.744331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:11.744898 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:11.745323 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:11.745341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.747953 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.747960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:11.749562 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:11.749585 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.751451 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.751488 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:11.751735 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:11.752419 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:11.752444 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:11.755581 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:11.760771 (+ 5190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:11.760775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:11.760778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:11.761348 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:11.761886 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:11.761892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:11.761896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:11.762092 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:11.762862 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:11.762867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:11.763041 (+ 174us) client_negotiation.cc:769] Sending connection context
0808 03:58:11.763219 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 03:58:11.763602 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":314,"threads_started":1}
I20260808 03:58:11.764235 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:11.728708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:38069 (local address 127.28.251.254:40631)
0808 03:58:11.743956 (+ 15248us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:11.743961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:11.743982 (+ 21us) server_negotiation.cc:407] Connection header received
0808 03:58:11.744044 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:11.744049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:11.744121 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:11.744255 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:11.745554 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.746805 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.750491 (+ 3686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:11.751204 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:11.751371 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:11.755880 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:11.755890 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:11.755895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:11.756462 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:11.760577 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:11.761504 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:11.761513 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:11.761518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:11.761745 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:11.762224 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:11.762231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:11.762235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:11.762711 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:11.762899 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:11.763370 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 03:58:11.763739 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15093,"thread_start_us":15668,"threads_started":1}
I20260808 03:58:11.765455 9011 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40631
I20260808 03:58:11.765602 9011 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:11.766086 9011 heartbeater.cc:511] Master 127.28.251.254:40631 requested a full tablet report, sending...
I20260808 03:58:11.766337 8721 ts_manager.cc:194] Registered new tserver with Master: 446ce0b231b841a2875981263d95cad2 (127.28.251.193:43373)
I20260808 03:58:11.766352 8419 ts_manager.cc:194] Registered new tserver with Master: 446ce0b231b841a2875981263d95cad2 (127.28.251.193:43373)
I20260808 03:58:11.767755 7435 ts_manager.cc:194] Re-registered known tserver with Master: 446ce0b231b841a2875981263d95cad2 (127.28.251.193:43373)
I20260808 03:58:11.768692 7435 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:38069
I20260808 03:58:11.855037 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 7754
I20260808 03:58:11.875756 8653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: NON_VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: true } } }
I20260808 03:58:11.876117 8653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:11.878152 9027 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516: attempting to promote NON_VOTER e4c1cfe150f14368b3815cbaf16832d4 to VOTER
I20260808 03:58:11.878778 9027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } }
I20260808 03:58:11.880889 8433 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 FOLLOWER]: Refusing update from remote peer 7b56826f1bab45dfb469a8b01ca93516: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 03:58:11.880950 8736 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 LEARNER]: Refusing update from remote peer 7b56826f1bab45dfb469a8b01ca93516: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 03:58:11.881578 8653 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 03:58:11.881983 9027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 03:58:11.884521 9027 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4c1cfe150f14368b3815cbaf16832d4 (127.28.251.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:11.885303 8433 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4c1cfe150f14368b3815cbaf16832d4 (127.28.251.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:11.888339 9027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:11.888612 9027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:11.888906 9027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:11.889153 9027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:11.889541 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:44113
--local_ip_for_outbound_sockets=127.28.251.194
--tserver_master_addrs=127.28.251.254:40631
--webserver_port=43011
--webserver_interface=127.28.251.194
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab
--principal=oryx/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.251.254:40631,127.28.251.253:35525
--tserver_master_addrs=127.28.251.254:40631,127.28.251.253:35525,127.28.251.252:42623 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 03:58:11.896106 9033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:11.897790 9033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 03:58:11.898598 9027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:11.898908 9027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:11.899516 8736 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4c1cfe150f14368b3815cbaf16832d4 (127.28.251.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:11.903510 8779 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b56826f1bab45dfb469a8b01ca93516" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:11.903790 8779 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 03:58:12.261523 9035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:12.261863 9035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:12.262014 9035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:12.269622 9035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:12.269790 9035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:12.269963 9035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:58:12.280503 9035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab
--principal=oryx/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:44113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=43011
--tserver_master_addrs=127.28.251.254:40631,127.28.251.253:35525,127.28.251.252:42623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:12.282052 9035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:12.283540 9035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:12.297919 9044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:12.297982 9042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:12.297947 9047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:12.299371 9035 server_base.cc:1034] running on GCE node
I20260808 03:58:12.300083 9035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:12.301133 9035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:12.302350 9035 hybrid_clock.cc:648] HybridClock initialized: now 1786161492302295 us; error 47 us; skew 500 ppm
I20260808 03:58:12.302706 9035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:12 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161492, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:12.306066 9035 init.cc:377] Logged in from keytab as oryx/127.28.251.194@KRBTEST.COM (short username oryx)
I20260808 03:58:12.307757 9035 webserver.cc:460] Webserver started at http://127.28.251.194:43011/ using document root <none> and password file <none>
I20260808 03:58:12.308478 9035 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:12.308598 9035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:12.314445 9035 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 03:58:12.317359 9064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:12.318749 9035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 03:58:12.318902 9035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "429e1d2b69e242f08e551d2021e5cba5"
format_stamp: "Formatted at 2026-08-08 03:58:08 on dist-test-slave-pww0"
server_key: "e1a0774cf85a8eb0ae487f9df58ec641"
server_key_iv: "5012381c1eab97cce34734d0ac2751e7"
server_key_version: "encryptionkey@0"
I20260808 03:58:12.319973 9035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:12.344432 9035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:12.349361 9035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:12.349648 9035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:12.350780 9035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:12.352349 9035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:12.352447 9035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:12.352531 9035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:12.352594 9035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:12.386747 9035 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:44113
I20260808 03:58:12.386860 9254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:44113 every 8 connection(s)
I20260808 03:58:12.388368 9035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:58:12.393416 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 9035
I20260808 03:58:12.393793 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 7983
Aug 08 03:58:12 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161492, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.253@KRBTEST.COM
Aug 08 03:58:12 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161492, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.252@KRBTEST.COM
Aug 08 03:58:12 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161492, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
Aug 08 03:58:12 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.251.195@KRBTEST.COM
I20260808 03:58:12.436650 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:12.391829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:55465 (local address 127.28.251.252:42623)
0808 03:58:12.400268 (+ 8439us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:12.400273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:12.415173 (+ 14900us) server_negotiation.cc:407] Connection header received
0808 03:58:12.415522 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:12.415529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:12.415620 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:12.415724 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:12.418693 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.419973 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.422816 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.423543 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.423645 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:12.429079 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:12.429086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:12.429092 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:12.429143 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:12.432644 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:12.433759 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:12.433765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:12.433769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:12.433946 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:12.434491 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:12.434497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:12.434500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:12.434817 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:12.434949 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:12.435996 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0808 03:58:12.436253 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8319,"thread_start_us":8180,"threads_started":1}
I20260808 03:58:12.436638 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:12.392195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:60927 (local address 127.28.251.253:35525)
0808 03:58:12.399878 (+ 7683us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:12.399887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:12.411670 (+ 11783us) server_negotiation.cc:407] Connection header received
0808 03:58:12.415750 (+ 4080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:12.415766 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:12.415873 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:12.415958 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:12.418997 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.420129 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.422845 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.423500 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.423609 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:12.429084 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:12.429092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:12.429097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:12.429140 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:12.432787 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:12.433964 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:12.433970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:12.433973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:12.434132 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:12.434540 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:12.434545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:12.434549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:12.434929 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:12.435079 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:12.435996 (+ 917us) server_negotiation.cc:299] Negotiation successful
0808 03:58:12.436248 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7530,"thread_start_us":7406,"threads_started":1}
I20260808 03:58:12.439056 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:12.394928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.28.251.195:34591)
0808 03:58:12.409087 (+ 14159us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:12.409098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:12.409141 (+ 43us) server_negotiation.cc:407] Connection header received
0808 03:58:12.409268 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:12.409281 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:12.409620 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:12.409854 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:12.417405 (+ 7551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.418896 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.420724 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.421588 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.421858 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:12.431446 (+ 9588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:12.431491 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:12.431513 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:12.431575 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:12.435370 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:12.436235 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:12.436243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:12.436246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:12.436394 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:12.436894 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:12.436899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:12.436902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:12.437351 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:12.437651 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:12.438344 (+ 693us) server_negotiation.cc:299] Negotiation successful
0808 03:58:12.438644 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13931,"thread_start_us":13708,"threads_started":1}
I20260808 03:58:12.439277 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:12.411652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.194:56323)
0808 03:58:12.415524 (+ 3872us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:12.415545 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:12.415736 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:12.418495 (+ 2759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:12.418501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:12.419130 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:12.419499 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:12.419512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.422085 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.422090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:12.423680 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:12.423695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.424908 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.424915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:12.425089 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:12.426310 (+ 1221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:12.426334 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:12.429341 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:12.435530 (+ 6189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:12.435535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:12.435538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:12.436034 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:12.437339 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:12.437344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:12.437348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:12.437541 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:12.438064 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:12.438069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:12.438241 (+ 172us) client_negotiation.cc:769] Sending connection context
0808 03:58:12.438405 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 03:58:12.438697 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3714,"thread_start_us":3526,"threads_started":1}
I20260808 03:58:12.439734 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:12.392435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:35525 (local address 127.28.251.194:60927)
0808 03:58:12.410855 (+ 18420us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:12.410916 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:12.414154 (+ 3238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:12.416209 (+ 2055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:12.416223 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:12.417367 (+ 1144us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:12.418798 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:12.418815 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.420337 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.420342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:12.422652 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:12.422676 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.424062 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.424072 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:12.424454 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:12.425311 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:12.425343 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:12.428725 (+ 3382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:12.432985 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:12.432991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:12.432995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:12.433815 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:12.434258 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:12.434263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:12.434267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:12.434429 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:12.435047 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:12.435050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:12.435494 (+ 444us) client_negotiation.cc:769] Sending connection context
0808 03:58:12.436557 (+ 1063us) client_negotiation.cc:240] Negotiation successful
0808 03:58:12.437614 (+ 1057us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18013,"mutex_wait_us":916,"spinlock_wait_cycles":18688,"thread_start_us":22100,"threads_started":1}
I20260808 03:58:12.441066 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:12.411998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:56323 (local address 127.28.251.254:40631)
0808 03:58:12.418136 (+ 6138us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:12.418142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:12.418167 (+ 25us) server_negotiation.cc:407] Connection header received
0808 03:58:12.418229 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:12.418234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:12.418312 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:12.418429 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:12.420709 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.421886 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.423881 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.424503 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.424626 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:12.429571 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:12.429578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:12.429582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:12.429629 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:12.433499 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:12.437051 (+ 3552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:12.437056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:12.437058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:12.437220 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:12.437658 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:12.437663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:12.437666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:12.437941 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:12.438080 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:12.440428 (+ 2348us) server_negotiation.cc:299] Negotiation successful
0808 03:58:12.440692 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6005,"thread_start_us":5897,"threads_started":1}
I20260808 03:58:12.441334 9259 heartbeater.cc:347] Connected to a master server at 127.28.251.253:35525
I20260808 03:58:12.441618 8185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:12.441677 9259 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:12.442250 9258 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40631
I20260808 03:58:12.442411 9258 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:12.442821 9259 heartbeater.cc:511] Master 127.28.251.253:35525 requested a full tablet report, sending...
I20260808 03:58:12.442978 9258 heartbeater.cc:511] Master 127.28.251.254:40631 requested a full tablet report, sending...
I20260808 03:58:12.444346 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:12.392435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:42623 (local address 127.28.251.194:55465)
0808 03:58:12.415036 (+ 22601us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:12.415061 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:12.415300 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:12.416209 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:12.416223 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:12.417367 (+ 1144us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:12.418458 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:12.418486 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.420217 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.420222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:12.422645 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:12.422660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:12.424061 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:12.424069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:12.424415 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:12.425448 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:12.425473 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:12.428803 (+ 3330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:12.432852 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:12.432864 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:12.432885 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:12.433625 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:12.434069 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:12.434073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:12.434076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:12.434346 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:12.434961 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:12.434970 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:12.435494 (+ 524us) client_negotiation.cc:769] Sending connection context
0808 03:58:12.437339 (+ 1845us) client_negotiation.cc:240] Negotiation successful
0808 03:58:12.438304 (+ 965us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22405,"mutex_wait_us":813,"spinlock_wait_cycles":92032,"thread_start_us":17716,"threads_started":1}
I20260808 03:58:12.444504 7435 ts_manager.cc:194] Re-registered known tserver with Master: 429e1d2b69e242f08e551d2021e5cba5 (127.28.251.194:44113)
I20260808 03:58:12.444669 8419 ts_manager.cc:194] Registered new tserver with Master: 429e1d2b69e242f08e551d2021e5cba5 (127.28.251.194:44113)
I20260808 03:58:12.445659 7435 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:56323
I20260808 03:58:12.448541 9256 heartbeater.cc:347] Connected to a master server at 127.28.251.252:42623
I20260808 03:58:12.448691 9256 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:12.449124 9256 heartbeater.cc:511] Master 127.28.251.252:42623 requested a full tablet report, sending...
I20260808 03:58:12.450505 8721 ts_manager.cc:194] Registered new tserver with Master: 429e1d2b69e242f08e551d2021e5cba5 (127.28.251.194:44113)
I20260808 03:58:12.533215 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 7983
I20260808 03:58:12.559985 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:34591
--local_ip_for_outbound_sockets=127.28.251.195
--tserver_master_addrs=127.28.251.254:40631
--webserver_port=34893
--webserver_interface=127.28.251.195
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab
--principal=oryx/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.251.254:40631,127.28.251.253:35525
--tserver_master_addrs=127.28.251.254:40631,127.28.251.253:35525,127.28.251.252:42623 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 03:58:12.772234 9011 heartbeater.cc:503] Master 127.28.251.254:40631 was elected leader, sending a full tablet report...
W20260808 03:58:12.920563 9278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:12.920931 9278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:12.921077 9278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:12.928890 9278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:12.929039 9278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:12.929195 9278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:58:12.939620 9278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab
--principal=oryx/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:34591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=34893
--tserver_master_addrs=127.28.251.254:40631,127.28.251.253:35525,127.28.251.252:42623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:12.941265 9278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:12.942657 9278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:12.957159 9287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:12.957191 9288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:12.957228 9290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:12.958042 9278 server_base.cc:1034] running on GCE node
I20260808 03:58:12.958798 9278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:12.959997 9278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:12.961246 9278 hybrid_clock.cc:648] HybridClock initialized: now 1786161492961182 us; error 54 us; skew 500 ppm
I20260808 03:58:12.961621 9278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:12 dist-test-slave-pww0 krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161492, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:12.965137 9278 init.cc:377] Logged in from keytab as oryx/127.28.251.195@KRBTEST.COM (short username oryx)
I20260808 03:58:12.966754 9278 webserver.cc:460] Webserver started at http://127.28.251.195:34893/ using document root <none> and password file <none>
I20260808 03:58:12.967576 9278 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:12.967718 9278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:12.974093 9278 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 03:58:12.977375 9297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:12.978893 9278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 03:58:12.979077 9278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "d37b8ac688c142a6b3b478912340a1e2"
format_stamp: "Formatted at 2026-08-08 03:58:09 on dist-test-slave-pww0"
server_key: "c186f16dbe595f1ce498786af3fa8466"
server_key_iv: "80fed0b7091c289e54e6123c3f54cdf5"
server_key_version: "encryptionkey@0"
I20260808 03:58:12.980285 9278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:12.996850 9278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:13.002202 9278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:13.002522 9278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:13.003791 9278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:13.005385 9278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:13.005487 9278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:13.005601 9278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:13.005686 9278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:13.039371 9278 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:34591
I20260808 03:58:13.039479 9498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:34591 every 8 connection(s)
I20260808 03:58:13.041038 9278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:58:13.048941 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 9278
Aug 08 03:58:13 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161492, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
Aug 08 03:58:13 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161492, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.252@KRBTEST.COM
Aug 08 03:58:13 dist-test-slave-pww0 krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161492, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.253@KRBTEST.COM
I20260808 03:58:13.075138 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.044378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:42295 (local address 127.28.251.252:42623)
0808 03:58:13.045346 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.045352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.046404 (+ 1052us) server_negotiation.cc:407] Connection header received
0808 03:58:13.049238 (+ 2834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.049245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.049344 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.049430 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:13.052719 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.053900 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.057333 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.058010 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.058125 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.065968 (+ 7843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:13.065976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:13.065981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:13.066028 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:13.069955 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.071306 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.071313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.071316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.071539 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.072218 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.072225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.072228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.072666 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:13.072843 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.074468 (+ 1625us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.074733 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":719,"threads_started":1}
I20260808 03:58:13.076375 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.047063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:49781 (local address 127.28.251.254:40631)
0808 03:58:13.047786 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.047791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.047810 (+ 19us) server_negotiation.cc:407] Connection header received
0808 03:58:13.049341 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.049347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.049440 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.049546 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:13.052732 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.054003 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.057357 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.058077 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.058189 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.065954 (+ 7765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:13.065963 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:13.065968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:13.066024 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:13.072329 (+ 6305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.073738 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.073745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.073748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.073931 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.074411 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.074417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.074421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.074811 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:13.074972 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.075769 (+ 797us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.076031 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":509,"threads_started":1}
I20260808 03:58:13.077658 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.046910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.195:49781)
0808 03:58:13.047673 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:13.047699 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:13.048863 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:13.049978 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:13.049991 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:13.051109 (+ 1118us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:13.052458 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.052484 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.054216 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.054223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.057129 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.057151 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.058627 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.058635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.059058 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.059959 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:13.060002 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:13.065671 (+ 5669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:13.072925 (+ 7254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:13.072930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:13.072934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:13.073568 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:13.074080 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:13.074085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:13.074088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:13.074257 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:13.075044 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:13.075049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:13.075216 (+ 167us) client_negotiation.cc:769] Sending connection context
0808 03:58:13.075403 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 03:58:13.076062 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":464,"spinlock_wait_cycles":3328,"thread_start_us":324,"threads_started":1}
I20260808 03:58:13.078822 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.045073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:42623 (local address 127.28.251.195:42295)
0808 03:58:13.046187 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:13.046237 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:13.048849 (+ 2612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:13.049976 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:13.049990 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:13.051108 (+ 1118us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:13.052447 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.052482 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.054149 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.054156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.057110 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.057123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.058594 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.058603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.059050 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.060208 (+ 1158us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:13.060226 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:13.064849 (+ 4623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:13.070235 (+ 5386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:13.070249 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:13.070273 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:13.071099 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:13.071709 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:13.071715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:13.071719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:13.072055 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:13.073373 (+ 1318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:13.073382 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:13.073942 (+ 560us) client_negotiation.cc:769] Sending connection context
0808 03:58:13.075403 (+ 1461us) client_negotiation.cc:240] Negotiation successful
0808 03:58:13.078380 (+ 2977us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":598,"spinlock_wait_cycles":147840,"thread_start_us":586,"threads_started":1}
I20260808 03:58:13.079954 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.044392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:48283 (local address 127.28.251.253:35525)
0808 03:58:13.044954 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.044961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.046388 (+ 1427us) server_negotiation.cc:407] Connection header received
0808 03:58:13.049238 (+ 2850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.049245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.049343 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.049430 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:13.052719 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.053900 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.057333 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.058020 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.058124 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.065543 (+ 7419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:13.065552 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:13.065557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:13.065607 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:13.072813 (+ 7206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.077442 (+ 4629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.077448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.077452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.077660 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.078165 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.078170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.078174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.078562 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:13.078705 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.079277 (+ 572us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.079577 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":716,"threads_started":1}
I20260808 03:58:13.080402 9500 heartbeater.cc:347] Connected to a master server at 127.28.251.252:42623
I20260808 03:58:13.080843 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.045073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:35525 (local address 127.28.251.195:48283)
0808 03:58:13.046188 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:13.046238 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:13.048843 (+ 2605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:13.049976 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:13.049990 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:13.051108 (+ 1118us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:13.052447 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.052482 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.054149 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.054156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.057101 (+ 2945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.057123 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.058593 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.058603 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.059613 (+ 1010us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.060093 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:13.060117 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:13.065243 (+ 5126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:13.072988 (+ 7745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:13.072993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:13.072996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:13.077233 (+ 4237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:13.077831 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:13.077835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:13.077838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:13.078019 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:13.078798 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:13.078802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:13.078996 (+ 194us) client_negotiation.cc:769] Sending connection context
0808 03:58:13.080145 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0808 03:58:13.080436 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":1158,"spinlock_wait_cycles":84224,"thread_start_us":437,"threads_started":1}
I20260808 03:58:13.080955 9500 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:13.081727 9502 heartbeater.cc:347] Connected to a master server at 127.28.251.253:35525
I20260808 03:58:13.081908 9502 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:13.082090 9500 heartbeater.cc:511] Master 127.28.251.252:42623 requested a full tablet report, sending...
I20260808 03:58:13.082355 9502 heartbeater.cc:511] Master 127.28.251.253:35525 requested a full tablet report, sending...
I20260808 03:58:13.084151 8419 ts_manager.cc:194] Registered new tserver with Master: d37b8ac688c142a6b3b478912340a1e2 (127.28.251.195:34591)
I20260808 03:58:13.084259 8721 ts_manager.cc:194] Registered new tserver with Master: d37b8ac688c142a6b3b478912340a1e2 (127.28.251.195:34591)
I20260808 03:58:13.085795 9501 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40631
I20260808 03:58:13.085944 9501 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:13.086422 9501 heartbeater.cc:511] Master 127.28.251.254:40631 requested a full tablet report, sending...
I20260808 03:58:13.087670 7435 ts_manager.cc:194] Re-registered known tserver with Master: d37b8ac688c142a6b3b478912340a1e2 (127.28.251.195:34591)
I20260808 03:58:13.089262 7435 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:49781
I20260808 03:58:13.097900 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.062889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.28.251.254:40631)
0808 03:58:13.076642 (+ 13753us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.076647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.076666 (+ 19us) server_negotiation.cc:407] Connection header received
0808 03:58:13.076715 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.076720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.076798 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.076891 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:13.082723 (+ 5832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.084107 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.086322 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.086942 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.087081 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.091359 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:13.091365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:13.091370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:13.091413 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:13.094825 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.095616 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.095623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.095627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.095769 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.096153 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.096159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.096163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.096574 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:13.096874 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.097221 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.097494 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13634,"thread_start_us":16777,"threads_started":1}
I20260808 03:58:13.102731 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.062761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.28.251.253:35525)
0808 03:58:13.063338 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.063345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.085323 (+ 21978us) server_negotiation.cc:407] Connection header received
0808 03:58:13.085389 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.085395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.085477 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.085573 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:13.088307 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.089542 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.091238 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.091937 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.092058 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.095306 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:13.095313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:13.095318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:13.095362 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:13.099097 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.100051 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.100057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.100060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.100257 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.100724 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.100730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.100733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.101116 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:13.101374 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.102035 (+ 661us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.102307 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":287,"threads_started":1}
I20260808 03:58:13.108052 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.076875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40380 (local address 127.28.251.252:42623)
0808 03:58:13.077067 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.077073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.091697 (+ 14624us) server_negotiation.cc:407] Connection header received
0808 03:58:13.091994 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.092001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.092114 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.092184 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:13.095035 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.096136 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.098019 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.098627 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.098770 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.102092 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:13.102098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:13.102104 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:13.102147 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:13.105324 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.105997 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.106003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.106007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.106181 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.106543 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.106548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.106552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.106854 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:13.107104 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.107498 (+ 394us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.107738 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 03:58:13.103142 7435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:58:13.110986 7435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:58:13.149700 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.135889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:43373 (local address 127.0.0.1:33126)
0808 03:58:13.139103 (+ 3214us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:13.139127 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:13.139288 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:13.140466 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:13.140471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:13.140499 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:13.140914 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.140931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.144122 (+ 3191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.144130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.146446 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.146464 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.148725 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.148732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.148967 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.149007 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 03:58:13.149109 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 03:58:13.149313 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3067,"thread_start_us":6079,"threads_started":1}
I20260808 03:58:13.149770 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.136086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.28.251.193:43373)
0808 03:58:13.137814 (+ 1728us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.137823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.139447 (+ 1624us) server_negotiation.cc:407] Connection header received
0808 03:58:13.139607 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.139616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.139955 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.140177 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:13.141149 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.143920 (+ 2771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.146659 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.148477 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.148678 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.148963 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.149176 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.149408 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1551,"thread_start_us":1432,"threads_started":1}
I20260808 03:58:13.153923 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.142376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:34591 (local address 127.0.0.1:40918)
0808 03:58:13.144354 (+ 1978us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:13.144376 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:13.144503 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:13.145639 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:13.145644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:13.145674 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:13.146119 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.146130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.147877 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.147882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.150265 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.150279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.152947 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.152954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.153156 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.153182 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 03:58:13.153281 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 03:58:13.153435 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1831,"thread_start_us":425,"threads_started":1}
I20260808 03:58:13.154175 8903 tablet_service.cc:1467] Processing CreateTablet for tablet ac4d4c2f1f5e440ea7aeb8c703392f5e (DEFAULT_TABLE table=test-table [id=210067d879c4487c9d053369042a9d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:13.154970 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.142771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.28.251.195:34591)
0808 03:58:13.143564 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.143572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.144709 (+ 1137us) server_negotiation.cc:407] Connection header received
0808 03:58:13.144804 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.144812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.145017 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.145209 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:13.146314 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.147647 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.151238 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.152725 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.152905 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.153090 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.154393 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.154580 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":396,"threads_started":1}
I20260808 03:58:13.156919 8903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4d4c2f1f5e440ea7aeb8c703392f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:13.158851 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.138538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.28.251.194:44113)
0808 03:58:13.151929 (+ 13391us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.151938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.151970 (+ 32us) server_negotiation.cc:407] Connection header received
0808 03:58:13.152071 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.152079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.152374 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.152603 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:13.153313 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.154714 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.156844 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.157802 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.157972 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.158168 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.158424 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.158594 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13210,"thread_start_us":13105,"threads_started":1}
I20260808 03:58:13.158864 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.138367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:44113 (local address 127.0.0.1:44324)
0808 03:58:13.142267 (+ 3900us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:13.142287 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:13.142401 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:13.152622 (+ 10221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:13.152627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:13.152665 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:13.153085 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.153099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.154855 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.154861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.156675 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.156690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.158039 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.158046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.158245 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.158276 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 03:58:13.158363 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 03:58:13.158503 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3762,"thread_start_us":404,"threads_started":1}
I20260808 03:58:13.159102 9382 tablet_service.cc:1467] Processing CreateTablet for tablet ac4d4c2f1f5e440ea7aeb8c703392f5e (DEFAULT_TABLE table=test-table [id=210067d879c4487c9d053369042a9d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:13.160830 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4d4c2f1f5e440ea7aeb8c703392f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:13.162880 9136 tablet_service.cc:1467] Processing CreateTablet for tablet ac4d4c2f1f5e440ea7aeb8c703392f5e (DEFAULT_TABLE table=test-table [id=210067d879c4487c9d053369042a9d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:13.164143 9136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4d4c2f1f5e440ea7aeb8c703392f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:13.174918 9557 tablet_bootstrap.cc:492] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: Bootstrap starting.
I20260808 03:58:13.177706 9558 tablet_bootstrap.cc:492] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: Bootstrap starting.
I20260808 03:58:13.179388 9557 tablet_bootstrap.cc:654] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:13.180464 9559 tablet_bootstrap.cc:492] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5: Bootstrap starting.
I20260808 03:58:13.180960 9557 log.cc:824] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:13.181850 9558 tablet_bootstrap.cc:654] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:13.183151 9558 log.cc:824] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:13.186852 9558 tablet_bootstrap.cc:492] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: No bootstrap required, opened a new log
I20260808 03:58:13.186923 9559 tablet_bootstrap.cc:654] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:13.187168 9558 ts_tablet_manager.cc:1397] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260808 03:58:13.188475 9559 log.cc:824] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:13.191303 9557 tablet_bootstrap.cc:492] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: No bootstrap required, opened a new log
I20260808 03:58:13.191668 9557 ts_tablet_manager.cc:1397] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.003s
I20260808 03:58:13.195833 9559 tablet_bootstrap.cc:492] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5: No bootstrap required, opened a new log
I20260808 03:58:13.196271 9559 ts_tablet_manager.cc:1397] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260808 03:58:13.198747 9558 raft_consensus.cc:348] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:13.199242 9558 raft_consensus.cc:374] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:13.199340 9558 raft_consensus.cc:729] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37b8ac688c142a6b3b478912340a1e2, State: Initialized, Role: FOLLOWER
I20260808 03:58:13.200112 9558 consensus_queue.cc:260] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:13.202252 9558 ts_tablet_manager.cc:1428] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: Time spent starting tablet: real 0.015s user 0.016s sys 0.001s
I20260808 03:58:13.202870 9501 heartbeater.cc:503] Master 127.28.251.254:40631 was elected leader, sending a full tablet report...
I20260808 03:58:13.202562 9557 raft_consensus.cc:348] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:13.203058 9557 raft_consensus.cc:374] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:13.203184 9557 raft_consensus.cc:729] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446ce0b231b841a2875981263d95cad2, State: Initialized, Role: FOLLOWER
I20260808 03:58:13.203981 9557 consensus_queue.cc:260] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:13.209028 9559 raft_consensus.cc:348] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:13.209748 9559 raft_consensus.cc:374] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:13.209905 9559 raft_consensus.cc:729] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429e1d2b69e242f08e551d2021e5cba5, State: Initialized, Role: FOLLOWER
I20260808 03:58:13.210374 9557 ts_tablet_manager.cc:1428] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20260808 03:58:13.211005 9559 consensus_queue.cc:260] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:13.214025 9258 heartbeater.cc:503] Master 127.28.251.254:40631 was elected leader, sending a full tablet report...
I20260808 03:58:13.214792 9559 ts_tablet_manager.cc:1428] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
W20260808 03:58:13.226809 9015 tablet.cc:2369] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:58:13.243870 9564 raft_consensus.cc:482] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:58:13.244212 9564 raft_consensus.cc:504] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:13.246016 9564 leader_election.cc:288] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 446ce0b231b841a2875981263d95cad2 (127.28.251.193:43373), 429e1d2b69e242f08e551d2021e5cba5 (127.28.251.194:44113)
I20260808 03:58:13.254160 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.246995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:50653 (local address 127.28.251.194:44113)
0808 03:58:13.247223 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.247229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.247248 (+ 19us) server_negotiation.cc:407] Connection header received
0808 03:58:13.247310 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.247313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.247379 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.247511 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:13.247844 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.248859 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.251426 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.252862 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.253058 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.253136 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.253709 (+ 573us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.253863 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 03:58:13.254151 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.246489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:59983 (local address 127.28.251.193:43373)
0808 03:58:13.246685 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.246690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.246713 (+ 23us) server_negotiation.cc:407] Connection header received
0808 03:58:13.246940 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.246945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.247010 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.247098 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:13.247478 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.248609 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.251414 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.252862 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.253053 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.253136 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.253709 (+ 573us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.253866 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 03:58:13.254220 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.246339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:44113 (local address 127.28.251.195:50653)
0808 03:58:13.246593 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:13.246616 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:13.246749 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:13.247534 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:13.247539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:13.247562 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:13.247703 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.247710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.248973 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.248979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.251205 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.251223 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.253100 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.253108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.253459 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.253525 (+ 66us) client_negotiation.cc:769] Sending connection context
0808 03:58:13.253665 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 03:58:13.253865 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":122}
I20260808 03:58:13.254218 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.246435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:43373 (local address 127.28.251.195:59983)
0808 03:58:13.246635 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:13.246654 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:13.246772 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:13.247128 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:13.247131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:13.247154 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:13.247311 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.247325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.248769 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.248773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.251203 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:13.251223 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.253100 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.253108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:13.253453 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.253525 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 03:58:13.253642 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 03:58:13.253865 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":193}
I20260808 03:58:13.255650 8940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d4c2f1f5e440ea7aeb8c703392f5e" candidate_uuid: "d37b8ac688c142a6b3b478912340a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446ce0b231b841a2875981263d95cad2" is_pre_election: true
I20260808 03:58:13.255650 9173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d4c2f1f5e440ea7aeb8c703392f5e" candidate_uuid: "d37b8ac688c142a6b3b478912340a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429e1d2b69e242f08e551d2021e5cba5" is_pre_election: true
I20260808 03:58:13.256282 9173 raft_consensus.cc:2454] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37b8ac688c142a6b3b478912340a1e2 in term 0.
I20260808 03:58:13.256282 8940 raft_consensus.cc:2454] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37b8ac688c142a6b3b478912340a1e2 in term 0.
I20260808 03:58:13.257162 9304 leader_election.cc:302] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 429e1d2b69e242f08e551d2021e5cba5, 446ce0b231b841a2875981263d95cad2, d37b8ac688c142a6b3b478912340a1e2; no voters:
I20260808 03:58:13.257661 9564 raft_consensus.cc:2781] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:58:13.257833 9564 raft_consensus.cc:482] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:58:13.257952 9564 raft_consensus.cc:3037] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:13.260377 9564 raft_consensus.cc:504] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:13.261075 9564 leader_election.cc:288] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [CANDIDATE]: Term 1 election: Requested vote from peers 446ce0b231b841a2875981263d95cad2 (127.28.251.193:43373), 429e1d2b69e242f08e551d2021e5cba5 (127.28.251.194:44113)
I20260808 03:58:13.261683 8940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d4c2f1f5e440ea7aeb8c703392f5e" candidate_uuid: "d37b8ac688c142a6b3b478912340a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446ce0b231b841a2875981263d95cad2"
I20260808 03:58:13.261693 9173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d4c2f1f5e440ea7aeb8c703392f5e" candidate_uuid: "d37b8ac688c142a6b3b478912340a1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429e1d2b69e242f08e551d2021e5cba5"
I20260808 03:58:13.261909 8940 raft_consensus.cc:3037] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:13.261906 9173 raft_consensus.cc:3037] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:13.264515 9173 raft_consensus.cc:2454] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37b8ac688c142a6b3b478912340a1e2 in term 1.
I20260808 03:58:13.264518 8940 raft_consensus.cc:2454] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37b8ac688c142a6b3b478912340a1e2 in term 1.
I20260808 03:58:13.265069 9303 leader_election.cc:302] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 429e1d2b69e242f08e551d2021e5cba5, 446ce0b231b841a2875981263d95cad2, d37b8ac688c142a6b3b478912340a1e2; no voters:
I20260808 03:58:13.265436 9564 raft_consensus.cc:2781] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:13.265920 9564 raft_consensus.cc:686] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 1 LEADER]: Becoming Leader. State: Replica: d37b8ac688c142a6b3b478912340a1e2, State: Running, Role: LEADER
I20260808 03:58:13.266464 9564 consensus_queue.cc:237] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:13.272145 7435 catalog_manager.cc:5495] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d37b8ac688c142a6b3b478912340a1e2 (127.28.251.195). New cstate: current_term: 1 leader_uuid: "d37b8ac688c142a6b3b478912340a1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } health_report { overall_health: HEALTHY } } }
W20260808 03:58:13.294302 9504 tablet.cc:2369] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:58:13.311789 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.305987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.28.251.195:34591)
0808 03:58:13.306228 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.306234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.306323 (+ 89us) server_negotiation.cc:407] Connection header received
0808 03:58:13.306576 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.306580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.306665 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.306809 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:58:13.307221 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.308058 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.309434 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.309866 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.309981 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.310552 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:58:13.311119 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:58:13.311274 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.311371 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.311493 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 03:58:13.320914 9173 raft_consensus.cc:1261] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5 [term 1 FOLLOWER]: Refusing update from remote peer d37b8ac688c142a6b3b478912340a1e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:13.320919 8940 raft_consensus.cc:1261] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2 [term 1 FOLLOWER]: Refusing update from remote peer d37b8ac688c142a6b3b478912340a1e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:13.321978 9569 consensus_queue.cc:1035] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:13.322346 9564 consensus_queue.cc:1035] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:13.347054 9581 mvcc.cc:205] Tried to move back new op lower bound from 7316117476631339008 to 7316117476424056832. Current Snapshot: MvccSnapshot[applied={T|T < 7316117476631339008}]
I20260808 03:58:13.358024 7435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:13.358350 7435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:13.365742 7435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b56826f1bab45dfb469a8b01ca93516: Sending DeleteTablet for 3 replicas of tablet ac4d4c2f1f5e440ea7aeb8c703392f5e
I20260808 03:58:13.367048 8903 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4d4c2f1f5e440ea7aeb8c703392f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:13 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:33126
I20260808 03:58:13.367633 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4d4c2f1f5e440ea7aeb8c703392f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:13 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:40918
I20260808 03:58:13.371434 9136 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4d4c2f1f5e440ea7aeb8c703392f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:13 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:44324
I20260808 03:58:13.422436 9598 tablet_replica.cc:330] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5: stopping tablet replica
I20260808 03:58:13.423187 9598 raft_consensus.cc:2229] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:13.424099 9598 raft_consensus.cc:2258] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:13.429057 9594 tablet_replica.cc:330] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: stopping tablet replica
I20260808 03:58:13.429750 9594 raft_consensus.cc:2229] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:13.430687 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 7371
I20260808 03:58:13.431502 9594 raft_consensus.cc:2258] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:13.433967 7490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:13.434104 9598 ts_tablet_manager.cc:1905] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:13.434311 9594 ts_tablet_manager.cc:1905] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:13.441640 9594 ts_tablet_manager.cc:1918] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:13.441984 9594 log.cc:1196] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/ac4d4c2f1f5e440ea7aeb8c703392f5e
I20260808 03:58:13.442423 9598 ts_tablet_manager.cc:1918] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:13.442618 9598 log.cc:1196] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/ac4d4c2f1f5e440ea7aeb8c703392f5e
I20260808 03:58:13.442811 9594 ts_tablet_manager.cc:1939] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 446ce0b231b841a2875981263d95cad2: Deleting consensus metadata
I20260808 03:58:13.443233 9598 ts_tablet_manager.cc:1939] T ac4d4c2f1f5e440ea7aeb8c703392f5e P 429e1d2b69e242f08e551d2021e5cba5: Deleting consensus metadata
I20260808 03:58:13.444083 9597 tablet_replica.cc:330] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: stopping tablet replica
I20260808 03:58:13.444628 9597 raft_consensus.cc:2229] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:58:13.445377 9597 raft_consensus.cc:2258] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:13.447804 9597 ts_tablet_manager.cc:1905] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:13.456720 9597 ts_tablet_manager.cc:1918] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:13.457336 9597 log.cc:1196] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/ac4d4c2f1f5e440ea7aeb8c703392f5e
I20260808 03:58:13.458031 9597 ts_tablet_manager.cc:1939] T ac4d4c2f1f5e440ea7aeb8c703392f5e P d37b8ac688c142a6b3b478912340a1e2: Deleting consensus metadata
I20260808 03:58:13.462723 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.378110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.28.251.253:35525)
0808 03:58:13.378347 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.378354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.425452 (+ 47098us) server_negotiation.cc:407] Connection header received
0808 03:58:13.425741 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.425747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.425854 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.425983 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:13.429456 (+ 3473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.430617 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.445625 (+ 15008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.446420 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.446548 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.455409 (+ 8861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:13.455418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:13.455423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:13.455510 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:13.459254 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.460330 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.460336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.460341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.460520 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.460983 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.460989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.460993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.461386 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:13.461676 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.462089 (+ 413us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.462360 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 03:58:13.480172 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.429539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.28.251.252:42623)
0808 03:58:13.429752 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.429758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.462883 (+ 33125us) server_negotiation.cc:407] Connection header received
0808 03:58:13.463061 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.463066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.463149 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.463234 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:13.465070 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.466277 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.468524 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.469197 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.469347 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.473327 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:13.473335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:13.473340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:13.473388 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:13.476939 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.477866 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.477872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.477876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.478043 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.478488 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.478493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.478497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.478899 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:13.479241 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.479535 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.479785 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 03:58:13.582023 7406 catalog_manager.cc:4841] TS 429e1d2b69e242f08e551d2021e5cba5 (127.28.251.194:44113): tablet ac4d4c2f1f5e440ea7aeb8c703392f5e (table test-table [id=210067d879c4487c9d053369042a9d22]) successfully deleted
I20260808 03:58:13.582254 7405 catalog_manager.cc:4841] TS 446ce0b231b841a2875981263d95cad2 (127.28.251.193:43373): tablet ac4d4c2f1f5e440ea7aeb8c703392f5e (table test-table [id=210067d879c4487c9d053369042a9d22]) successfully deleted
W20260808 03:58:13.582422 7405 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.251.193:43373, user_credentials={real_user=slave}} blocked reactor thread for 137285us
I20260808 03:58:13.582922 7405 catalog_manager.cc:4841] TS d37b8ac688c142a6b3b478912340a1e2 (127.28.251.195:34591): tablet ac4d4c2f1f5e440ea7aeb8c703392f5e (table test-table [id=210067d879c4487c9d053369042a9d22]) successfully deleted
I20260808 03:58:13.583189 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 7371
I20260808 03:58:13.635551 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.619020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.28.251.253:35525)
0808 03:58:13.619217 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.619223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.619866 (+ 643us) server_negotiation.cc:407] Connection header received
0808 03:58:13.620062 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.620069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.620178 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.620300 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:13.623013 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.624029 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.625605 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.626225 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.626370 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.629547 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:13.629554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:13.629559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:13.629610 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:13.632809 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.633484 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.633491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.633495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.633669 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.634081 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.634086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.634090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.634374 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:13.634618 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.634952 (+ 334us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.635177 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 03:58:13.639106 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:13.621572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.28.251.252:42623)
0808 03:58:13.621817 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:13.621823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:13.621842 (+ 19us) server_negotiation.cc:407] Connection header received
0808 03:58:13.621910 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:13.621916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:13.621998 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:13.622093 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:13.625327 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.626139 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.627943 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:13.628628 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:13.628786 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:13.631843 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:13.631851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:13.631855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:13.631903 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:13.636005 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.636883 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.636890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.636893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.637065 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:13.637541 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:13.637547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:13.637551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:13.637880 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:13.638136 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:13.638526 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 03:58:13.638783 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 03:58:14.347431 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.346435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.194:36159)
0808 03:58:14.347021 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.347199 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":291,"threads_started":1}
W20260808 03:58:14.348177 9258 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:40631 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
I20260808 03:58:14.350418 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.348905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.195:45911)
0808 03:58:14.349890 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.350078 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":674,"threads_started":1}
W20260808 03:58:14.351192 9501 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:40631 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
I20260808 03:58:14.354087 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.353333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.193:53755)
0808 03:58:14.353785 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.353917 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":240,"threads_started":1}
W20260808 03:58:14.354478 9011 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:40631 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
I20260808 03:58:14.866932 9612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b56826f1bab45dfb469a8b01ca93516)
I20260808 03:58:14.867259 9612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } }
I20260808 03:58:14.867841 9613 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b56826f1bab45dfb469a8b01ca93516)
I20260808 03:58:14.868198 9613 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } }
I20260808 03:58:14.868912 9612 leader_election.cc:288] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b56826f1bab45dfb469a8b01ca93516 (127.28.251.254:40631), e4c1cfe150f14368b3815cbaf16832d4 (127.28.251.252:42623)
I20260808 03:58:14.869562 9613 leader_election.cc:288] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b56826f1bab45dfb469a8b01ca93516 (127.28.251.254:40631), 548398cfe4c049bb8bcbc1e169424ca8 (127.28.251.253:35525)
I20260808 03:58:14.882584 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.869933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51096)
0808 03:58:14.881910 (+ 11977us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.882252 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11814,"threads_started":1}
W20260808 03:58:14.885177 8691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
I20260808 03:58:14.886737 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.869174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51090)
0808 03:58:14.886108 (+ 16934us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.886371 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16808,"thread_start_us":15317,"threads_started":1}
W20260808 03:58:14.887383 8392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
I20260808 03:58:14.888445 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.887775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51112)
0808 03:58:14.887975 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.888192 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 03:58:14.888831 8691 leader_election.cc:334] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b56826f1bab45dfb469a8b01ca93516 (127.28.251.254:40631): Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
I20260808 03:58:14.890697 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.890210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51128)
0808 03:58:14.890418 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.890526 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260808 03:58:14.891055 8392 leader_election.cc:334] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b56826f1bab45dfb469a8b01ca93516 (127.28.251.254:40631): Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
I20260808 03:58:14.892227 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.870026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:35525 (local address 127.0.0.1:41690)
0808 03:58:14.882990 (+ 12964us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.883016 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:14.883327 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:14.884072 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:14.884077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:14.884110 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:14.884391 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:14.884406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.887688 (+ 3282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.887694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:14.889919 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:14.889938 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.891423 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.891429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:14.891634 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:14.891661 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 03:58:14.891751 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 03:58:14.891904 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12847,"thread_start_us":15318,"threads_started":1}
I20260808 03:58:14.892299 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.870141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.28.251.253:35525)
0808 03:58:14.881273 (+ 11132us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:14.881282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:14.883189 (+ 1907us) server_negotiation.cc:407] Connection header received
0808 03:58:14.883546 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:14.883552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:14.883679 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:14.883811 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:14.885564 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.887054 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.890069 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.891255 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.891422 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:14.891531 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:14.891804 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 03:58:14.891945 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10904,"thread_start_us":11090,"threads_started":1}
I20260808 03:58:14.893000 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4c1cfe150f14368b3815cbaf16832d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "548398cfe4c049bb8bcbc1e169424ca8" is_pre_election: true
I20260808 03:58:14.893322 8433 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c1cfe150f14368b3815cbaf16832d4 in term 1.
I20260808 03:58:14.893859 8693 leader_election.cc:302] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548398cfe4c049bb8bcbc1e169424ca8, e4c1cfe150f14368b3815cbaf16832d4; no voters: 7b56826f1bab45dfb469a8b01ca93516
I20260808 03:58:14.894265 9613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 03:58:14.894434 9613 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7b56826f1bab45dfb469a8b01ca93516)
I20260808 03:58:14.894557 9613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 03:58:14.894598 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.885331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:42623 (local address 127.0.0.1:40416)
0808 03:58:14.886269 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.886285 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:14.886412 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:14.888147 (+ 1735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:14.888153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:14.888188 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:14.888643 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:14.888656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.890832 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.890836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:14.892571 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:14.892593 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.893828 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.893834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:14.894004 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:14.894031 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 03:58:14.894130 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 03:58:14.894270 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":1704,"threads_started":1}
I20260808 03:58:14.894661 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.886942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.28.251.252:42623)
0808 03:58:14.887772 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:14.887777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:14.887804 (+ 27us) server_negotiation.cc:407] Connection header received
0808 03:58:14.887868 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:14.887873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:14.887952 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:14.889310 (+ 1358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:14.889668 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.890691 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.892757 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.893644 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.893811 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:14.893889 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:14.894171 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 03:58:14.894312 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":537,"threads_started":1}
I20260808 03:58:14.895399 8736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "548398cfe4c049bb8bcbc1e169424ca8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4c1cfe150f14368b3815cbaf16832d4" is_pre_election: true
I20260808 03:58:14.897001 9613 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } }
I20260808 03:58:14.897509 8736 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548398cfe4c049bb8bcbc1e169424ca8 in current term 2: Already voted for candidate e4c1cfe150f14368b3815cbaf16832d4 in this term.
I20260808 03:58:14.897781 9613 leader_election.cc:288] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [CANDIDATE]: Term 2 election: Requested vote from peers 7b56826f1bab45dfb469a8b01ca93516 (127.28.251.254:40631), 548398cfe4c049bb8bcbc1e169424ca8 (127.28.251.253:35525)
I20260808 03:58:14.898416 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.897891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51134)
0808 03:58:14.898154 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.898246 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260808 03:58:14.898471 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4c1cfe150f14368b3815cbaf16832d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "548398cfe4c049bb8bcbc1e169424ca8"
I20260808 03:58:14.898650 8392 leader_election.cc:302] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548398cfe4c049bb8bcbc1e169424ca8; no voters: 7b56826f1bab45dfb469a8b01ca93516, e4c1cfe150f14368b3815cbaf16832d4
I20260808 03:58:14.898725 8433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 03:58:14.899700 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.899257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51138)
0808 03:58:14.899424 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.899557 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 03:58:14.899896 8691 leader_election.cc:334] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7b56826f1bab45dfb469a8b01ca93516 (127.28.251.254:40631): Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
I20260808 03:58:14.901700 8433 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c1cfe150f14368b3815cbaf16832d4 in term 2.
I20260808 03:58:14.901919 9612 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 03:58:14.902197 8693 leader_election.cc:302] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548398cfe4c049bb8bcbc1e169424ca8, e4c1cfe150f14368b3815cbaf16832d4; no voters: 7b56826f1bab45dfb469a8b01ca93516
I20260808 03:58:14.902633 9613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 03:58:14.903155 9613 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [term 2 LEADER]: Becoming Leader. State: Replica: e4c1cfe150f14368b3815cbaf16832d4, State: Running, Role: LEADER
I20260808 03:58:14.903681 9613 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } }
I20260808 03:58:14.905663 9631 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4c1cfe150f14368b3815cbaf16832d4. Latest consensus state: current_term: 2 leader_uuid: "e4c1cfe150f14368b3815cbaf16832d4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:14.906128 9631 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:14.906829 9634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:58:14.911103 9634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:58:14.912290 9634 catalog_manager.cc:1259] Loaded cluster ID: 66b7c2628b5a49eb9ab7cf57e481cb60
I20260808 03:58:14.912388 9634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:58:14.913715 9634 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:58:14.914588 9634 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: Loaded TSK: 0
I20260808 03:58:14.915661 9634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 03:58:14.939956 8721 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:58:14.947324 8721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:58:14.952351 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.951872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51158)
0808 03:58:14.952074 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.952187 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 03:58:14.952461 8433 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [term 2 FOLLOWER]: Refusing update from remote peer e4c1cfe150f14368b3815cbaf16832d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 03:58:14.953445 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.953023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51168)
0808 03:58:14.953210 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.953305 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 03:58:14.953361 9631 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260808 03:58:14.953679 8691 consensus_peers.cc:487] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 -> Peer 7b56826f1bab45dfb469a8b01ca93516 (127.28.251.254:40631): Couldn't send request to peer 7b56826f1bab45dfb469a8b01ca93516. Status: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 03:58:14.956579 9612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4c1cfe150f14368b3815cbaf16832d4. Latest consensus state: current_term: 2 leader_uuid: "e4c1cfe150f14368b3815cbaf16832d4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:14.956939 9612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 03:58:14.958583 9631 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e4c1cfe150f14368b3815cbaf16832d4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:14.958940 9631 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:14.959616 9612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e4c1cfe150f14368b3815cbaf16832d4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b56826f1bab45dfb469a8b01ca93516" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40631 } } peers { permanent_uuid: "548398cfe4c049bb8bcbc1e169424ca8" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 35525 } attrs { promote: false } } peers { permanent_uuid: "e4c1cfe150f14368b3815cbaf16832d4" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 42623 } attrs { promote: false } } }
I20260808 03:58:14.959944 9612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548398cfe4c049bb8bcbc1e169424ca8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 03:58:14.960152 9637 mvcc.cc:205] Tried to move back new op lower bound from 7316117483316572160 to 7316117483129012224. Current Snapshot: MvccSnapshot[applied={T|T < 7316117483316572160}]
I20260808 03:58:14.978844 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.968279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:44113 (local address 127.0.0.1:44332)
0808 03:58:14.968497 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.968517 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:14.968628 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:14.969425 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:14.969430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:14.969455 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:14.969677 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:14.969689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.971413 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.971420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:14.973716 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:14.973744 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.977700 (+ 3956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.977709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:14.978036 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:14.978082 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 03:58:14.978174 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 03:58:14.978387 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":44}
I20260808 03:58:14.980526 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.968671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:43373 (local address 127.0.0.1:33136)
0808 03:58:14.968931 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.968948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:14.969054 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:14.969648 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:14.969652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:14.969675 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:14.970036 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:14.970050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.971655 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.971662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:14.975732 (+ 4070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:14.975754 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.977701 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.977709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:14.977995 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:14.978042 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 03:58:14.978129 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 03:58:14.980083 (+ 1954us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":3329}
I20260808 03:58:14.980897 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.968828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.28.251.193:43373)
0808 03:58:14.969277 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:14.969310 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:14.969332 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:58:14.969406 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:14.969411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:14.969489 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:14.970192 (+ 703us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:14.970387 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.971483 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.975964 (+ 4481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.977355 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.980229 (+ 2874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:14.980354 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:14.980427 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 03:58:14.980587 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":242,"threads_started":1}
I20260808 03:58:14.981848 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.968479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.28.251.194:44113)
0808 03:58:14.969065 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:14.969071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:14.969090 (+ 19us) server_negotiation.cc:407] Connection header received
0808 03:58:14.969165 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:14.969170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:14.969247 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:14.969408 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:14.969976 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.971207 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.976070 (+ 4863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.977453 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.981092 (+ 3639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:14.981240 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:14.981309 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 03:58:14.981508 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":345,"threads_started":1}
I20260808 03:58:14.982748 8903 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1cb3483b8144068d7e149645d653f2 (DEFAULT_TABLE table=test-table [id=c20c69f0281b4a86a7dea14ceaee6d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:14.983279 8903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1cb3483b8144068d7e149645d653f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:14.986830 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.969908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:34591 (local address 127.0.0.1:40944)
0808 03:58:14.979179 (+ 9271us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:14.979202 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:14.979378 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:14.981427 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:14.981433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:14.981463 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:14.981678 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:14.981693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.983043 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.983048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:14.984538 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:14.984556 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.986047 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.986053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:14.986243 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:14.986271 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 03:58:14.986360 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 03:58:14.986518 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9144,"thread_start_us":11780,"threads_started":1}
I20260808 03:58:14.986902 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:14.970034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.28.251.195:34591)
0808 03:58:14.980848 (+ 10814us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:14.980854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:14.980876 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:58:14.980956 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:14.980961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:14.981057 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:14.981380 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:14.981902 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.982872 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.984751 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:14.985873 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:14.986047 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:14.986130 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:14.986427 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 03:58:14.986584 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10675,"thread_start_us":11181,"threads_started":1}
I20260808 03:58:14.988000 9382 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1cb3483b8144068d7e149645d653f2 (DEFAULT_TABLE table=test-table [id=c20c69f0281b4a86a7dea14ceaee6d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:14.988519 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1cb3483b8144068d7e149645d653f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:14.989732 9651 tablet_bootstrap.cc:492] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2: Bootstrap starting.
I20260808 03:58:14.989750 9136 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1cb3483b8144068d7e149645d653f2 (DEFAULT_TABLE table=test-table [id=c20c69f0281b4a86a7dea14ceaee6d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:14.990240 9136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1cb3483b8144068d7e149645d653f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:14.991420 9651 tablet_bootstrap.cc:654] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:14.998036 9651 tablet_bootstrap.cc:492] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2: No bootstrap required, opened a new log
I20260808 03:58:14.998210 9651 ts_tablet_manager.cc:1397] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 03:58:14.998262 9653 tablet_bootstrap.cc:492] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2: Bootstrap starting.
I20260808 03:58:14.999114 9651 raft_consensus.cc:348] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:14.999434 9651 raft_consensus.cc:374] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:14.999554 9651 raft_consensus.cc:729] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446ce0b231b841a2875981263d95cad2, State: Initialized, Role: FOLLOWER
I20260808 03:58:14.999969 9651 consensus_queue.cc:260] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:15.000566 9653 tablet_bootstrap.cc:654] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:15.002467 9651 ts_tablet_manager.cc:1428] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 03:58:15.004860 9653 tablet_bootstrap.cc:492] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2: No bootstrap required, opened a new log
I20260808 03:58:15.004914 9655 tablet_bootstrap.cc:492] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5: Bootstrap starting.
I20260808 03:58:15.005286 9653 ts_tablet_manager.cc:1397] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260808 03:58:15.005684 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.003243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.193:60487)
0808 03:58:15.005214 (+ 1971us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.005395 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1810,"thread_start_us":1745,"threads_started":1}
I20260808 03:58:15.006449 9007 heartbeater.cc:503] Master 127.28.251.252:42623 was elected leader, sending a full tablet report...
I20260808 03:58:15.006336 9653 raft_consensus.cc:348] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:15.006592 9653 raft_consensus.cc:374] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:15.006752 9653 raft_consensus.cc:729] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37b8ac688c142a6b3b478912340a1e2, State: Initialized, Role: FOLLOWER
I20260808 03:58:15.007073 9653 consensus_queue.cc:260] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:15.007694 9655 tablet_bootstrap.cc:654] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:15.008196 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.007646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.193:60027)
0808 03:58:15.007899 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.007997 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260808 03:58:15.011178 9653 ts_tablet_manager.cc:1428] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260808 03:58:15.013027 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.011999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.195:58541)
0808 03:58:15.012591 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.012732 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":323,"threads_started":1}
I20260808 03:58:15.013789 9500 heartbeater.cc:503] Master 127.28.251.252:42623 was elected leader, sending a full tablet report...
I20260808 03:58:15.014480 9655 tablet_bootstrap.cc:492] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5: No bootstrap required, opened a new log
I20260808 03:58:15.014490 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.014043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.195:56317)
0808 03:58:15.014220 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.014311 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 03:58:15.014648 9655 ts_tablet_manager.cc:1397] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260808 03:58:15.015448 9655 raft_consensus.cc:348] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:15.015738 9655 raft_consensus.cc:374] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:15.015841 9655 raft_consensus.cc:729] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429e1d2b69e242f08e551d2021e5cba5, State: Initialized, Role: FOLLOWER
I20260808 03:58:15.016074 9655 consensus_queue.cc:260] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:15.016901 9655 ts_tablet_manager.cc:1428] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 03:58:15.018679 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.017362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.194:51051)
0808 03:58:15.018239 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.018418 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":604,"threads_started":1}
I20260808 03:58:15.019819 9256 heartbeater.cc:503] Master 127.28.251.252:42623 was elected leader, sending a full tablet report...
I20260808 03:58:15.020004 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.019371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.194:44807)
0808 03:58:15.019702 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.019798 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 03:58:15.150218 9260 tablet.cc:2369] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:58:15.245587 9656 raft_consensus.cc:482] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:58:15.245847 9656 raft_consensus.cc:504] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:15.247493 9656 leader_election.cc:288] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429e1d2b69e242f08e551d2021e5cba5 (127.28.251.194:44113), d37b8ac688c142a6b3b478912340a1e2 (127.28.251.195:34591)
I20260808 03:58:15.253806 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.247814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:34591 (local address 127.28.251.193:41847)
0808 03:58:15.248043 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.248066 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:15.248228 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:15.248592 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:15.248597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:15.248628 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:15.248989 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:15.249000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.250077 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.250080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:15.251960 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:15.251978 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.253203 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.253208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:15.253348 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:15.253376 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 03:58:15.253452 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 03:58:15.253567 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260808 03:58:15.253952 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.247974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:41847 (local address 127.28.251.195:34591)
0808 03:58:15.248197 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:15.248203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:15.248227 (+ 24us) server_negotiation.cc:407] Connection header received
0808 03:58:15.248384 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:15.248390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:15.248460 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:15.248540 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:15.249167 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.249949 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.252145 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.253001 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.253180 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:15.253272 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:15.253508 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 03:58:15.253659 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 03:58:15.254966 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.248057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:59553 (local address 127.28.251.194:44113)
0808 03:58:15.248256 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:15.248261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:15.249175 (+ 914us) server_negotiation.cc:407] Connection header received
0808 03:58:15.249367 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:15.249371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:15.249454 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:15.249531 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:15.250081 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.250909 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.252947 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.254096 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.254263 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:15.254349 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:15.254646 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 03:58:15.254763 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 03:58:15.255106 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.247814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:44113 (local address 127.28.251.193:59553)
0808 03:58:15.249050 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.249066 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:15.249227 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:15.249567 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:15.249571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:15.249597 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:15.249909 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:15.249918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.251063 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.251069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:15.252759 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:15.252772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.254329 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.254335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:15.254479 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:15.254504 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 03:58:15.254586 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 03:58:15.254716 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"mutex_wait_us":530,"thread_start_us":338,"threads_started":1}
I20260808 03:58:15.255009 9415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1cb3483b8144068d7e149645d653f2" candidate_uuid: "446ce0b231b841a2875981263d95cad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37b8ac688c142a6b3b478912340a1e2" is_pre_election: true
I20260808 03:58:15.255542 9415 raft_consensus.cc:2454] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446ce0b231b841a2875981263d95cad2 in term 0.
I20260808 03:58:15.255906 9173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1cb3483b8144068d7e149645d653f2" candidate_uuid: "446ce0b231b841a2875981263d95cad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429e1d2b69e242f08e551d2021e5cba5" is_pre_election: true
I20260808 03:58:15.256188 9173 raft_consensus.cc:2454] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446ce0b231b841a2875981263d95cad2 in term 0.
I20260808 03:58:15.256188 8833 leader_election.cc:302] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446ce0b231b841a2875981263d95cad2, d37b8ac688c142a6b3b478912340a1e2; no voters:
I20260808 03:58:15.256646 9656 raft_consensus.cc:2781] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:58:15.256763 9656 raft_consensus.cc:482] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:58:15.256873 9656 raft_consensus.cc:3037] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:15.259114 9656 raft_consensus.cc:504] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:15.259810 9656 leader_election.cc:288] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [CANDIDATE]: Term 1 election: Requested vote from peers 429e1d2b69e242f08e551d2021e5cba5 (127.28.251.194:44113), d37b8ac688c142a6b3b478912340a1e2 (127.28.251.195:34591)
I20260808 03:58:15.260382 9173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1cb3483b8144068d7e149645d653f2" candidate_uuid: "446ce0b231b841a2875981263d95cad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429e1d2b69e242f08e551d2021e5cba5"
I20260808 03:58:15.260406 9415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1cb3483b8144068d7e149645d653f2" candidate_uuid: "446ce0b231b841a2875981263d95cad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37b8ac688c142a6b3b478912340a1e2"
I20260808 03:58:15.260582 9173 raft_consensus.cc:3037] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:15.260605 9415 raft_consensus.cc:3037] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:15.262737 9173 raft_consensus.cc:2454] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446ce0b231b841a2875981263d95cad2 in term 1.
I20260808 03:58:15.262778 9415 raft_consensus.cc:2454] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446ce0b231b841a2875981263d95cad2 in term 1.
I20260808 03:58:15.263269 8835 leader_election.cc:302] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 429e1d2b69e242f08e551d2021e5cba5, 446ce0b231b841a2875981263d95cad2; no voters:
I20260808 03:58:15.263635 9656 raft_consensus.cc:2781] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:15.264123 9656 raft_consensus.cc:686] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 1 LEADER]: Becoming Leader. State: Replica: 446ce0b231b841a2875981263d95cad2, State: Running, Role: LEADER
I20260808 03:58:15.264566 9656 consensus_queue.cc:237] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } } peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } }
I20260808 03:58:15.265547 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.265061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.193:48487)
0808 03:58:15.265249 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.265379 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 03:58:15.268591 8721 catalog_manager.cc:5495] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 446ce0b231b841a2875981263d95cad2 (127.28.251.193). New cstate: current_term: 1 leader_uuid: "446ce0b231b841a2875981263d95cad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446ce0b231b841a2875981263d95cad2" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 } health_report { overall_health: UNKNOWN } } }
I20260808 03:58:15.351397 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.345772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.28.251.193:43373)
0808 03:58:15.346003 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:15.346009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:15.346069 (+ 60us) server_negotiation.cc:407] Connection header received
0808 03:58:15.346335 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:15.346340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:15.346438 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:15.346578 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:58:15.346973 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.347736 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.349098 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.349589 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.349704 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:15.350150 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:58:15.350752 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:58:15.350893 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:15.351016 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 03:58:15.351108 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 03:58:15.356144 9173 raft_consensus.cc:1261] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5 [term 1 FOLLOWER]: Refusing update from remote peer 446ce0b231b841a2875981263d95cad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:15.356606 9415 raft_consensus.cc:1261] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2 [term 1 FOLLOWER]: Refusing update from remote peer 446ce0b231b841a2875981263d95cad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:15.356926 9670 consensus_queue.cc:1035] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429e1d2b69e242f08e551d2021e5cba5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 44113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:15.357455 9656 consensus_queue.cc:1035] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37b8ac688c142a6b3b478912340a1e2" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:15.360852 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.360330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.195:35953)
0808 03:58:15.360515 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.360644 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 03:58:15.361717 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.359927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.194:37169)
0808 03:58:15.361393 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.361527 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1345}
I20260808 03:58:15.365417 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.364892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.194:33645)
0808 03:58:15.365132 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.365228 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260808 03:58:15.378069 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.377301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.193:40851)
0808 03:58:15.377758 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.377862 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336}
I20260808 03:58:15.378178 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.377675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.195:53591)
0808 03:58:15.377957 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.378039 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260808 03:58:15.393327 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.392494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.28.251.193:53709)
0808 03:58:15.393006 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.393140 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385}
I20260808 03:58:15.395728 8721 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:15.396023 8721 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:15.404214 8721 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4c1cfe150f14368b3815cbaf16832d4: Sending DeleteTablet for 3 replicas of tablet 3e1cb3483b8144068d7e149645d653f2
I20260808 03:58:15.405288 9136 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1cb3483b8144068d7e149645d653f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:15 UTC) from {username='oryx', principal='oryx/127.28.251.252@KRBTEST.COM'} at 127.0.0.1:44332
I20260808 03:58:15.406067 9691 tablet_replica.cc:330] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5: stopping tablet replica
I20260808 03:58:15.406217 9382 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1cb3483b8144068d7e149645d653f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:15 UTC) from {username='oryx', principal='oryx/127.28.251.252@KRBTEST.COM'} at 127.0.0.1:40944
I20260808 03:58:15.406291 9691 raft_consensus.cc:2229] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:15.406677 9691 raft_consensus.cc:2258] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:15.406984 8903 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1cb3483b8144068d7e149645d653f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:15 UTC) from {username='oryx', principal='oryx/127.28.251.252@KRBTEST.COM'} at 127.0.0.1:33136
I20260808 03:58:15.408376 9691 ts_tablet_manager.cc:1905] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:15.410389 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 8699
I20260808 03:58:15.412658 9694 tablet_replica.cc:330] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2: stopping tablet replica
I20260808 03:58:15.412884 9694 raft_consensus.cc:2229] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:15.415309 9694 raft_consensus.cc:2258] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:15.416811 9694 ts_tablet_manager.cc:1905] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:15.421787 9696 tablet_replica.cc:330] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2: stopping tablet replica
I20260808 03:58:15.422098 9696 raft_consensus.cc:2229] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:58:15.422580 9691 ts_tablet_manager.cc:1918] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:15.422756 9691 log.cc:1196] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/3e1cb3483b8144068d7e149645d653f2
I20260808 03:58:15.422806 9696 raft_consensus.cc:2258] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:15.424029 9696 ts_tablet_manager.cc:1905] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:15.424152 9691 ts_tablet_manager.cc:1939] T 3e1cb3483b8144068d7e149645d653f2 P 429e1d2b69e242f08e551d2021e5cba5: Deleting consensus metadata
I20260808 03:58:15.428830 9694 ts_tablet_manager.cc:1918] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:15.428995 9694 log.cc:1196] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/3e1cb3483b8144068d7e149645d653f2
I20260808 03:58:15.429442 9694 ts_tablet_manager.cc:1939] T 3e1cb3483b8144068d7e149645d653f2 P d37b8ac688c142a6b3b478912340a1e2: Deleting consensus metadata
I20260808 03:58:15.430922 8692 catalog_manager.cc:4841] TS d37b8ac688c142a6b3b478912340a1e2 (127.28.251.195:34591): tablet 3e1cb3483b8144068d7e149645d653f2 (table test-table [id=c20c69f0281b4a86a7dea14ceaee6d77]) successfully deleted
I20260808 03:58:15.431025 8693 catalog_manager.cc:4841] TS 429e1d2b69e242f08e551d2021e5cba5 (127.28.251.194:44113): tablet 3e1cb3483b8144068d7e149645d653f2 (table test-table [id=c20c69f0281b4a86a7dea14ceaee6d77]) successfully deleted
I20260808 03:58:15.432323 9696 ts_tablet_manager.cc:1918] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:15.432477 9696 log.cc:1196] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/3e1cb3483b8144068d7e149645d653f2
I20260808 03:58:15.432917 9696 ts_tablet_manager.cc:1939] T 3e1cb3483b8144068d7e149645d653f2 P 446ce0b231b841a2875981263d95cad2: Deleting consensus metadata
I20260808 03:58:15.434017 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.411295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.28.251.193:43373)
0808 03:58:15.411680 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:15.411688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:15.412920 (+ 1232us) server_negotiation.cc:407] Connection header received
0808 03:58:15.413205 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:15.413212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:15.413342 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:15.413443 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:15.415793 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.417164 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.419354 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.420409 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.420527 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:15.425144 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:15.425165 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:15.425186 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:15.425247 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:15.429835 (+ 4588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:15.430954 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:15.430965 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:15.430969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:15.431159 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:15.431781 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:15.431788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:15.431792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:15.432353 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:15.432732 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:15.433392 (+ 660us) server_negotiation.cc:299] Negotiation successful
0808 03:58:15.433644 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260808 03:58:15.434092 8692 catalog_manager.cc:4841] TS 446ce0b231b841a2875981263d95cad2 (127.28.251.193:43373): tablet 3e1cb3483b8144068d7e149645d653f2 (table test-table [id=c20c69f0281b4a86a7dea14ceaee6d77]) successfully deleted
I20260808 03:58:15.434763 9000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:15.470227 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.469772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51180)
0808 03:58:15.469969 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.470069 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 03:58:15.471732 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.471300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51190)
0808 03:58:15.471512 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.471598 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 03:58:15.540969 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 8699
I20260808 03:58:15.569470 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 9035
I20260808 03:58:15.583250 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.570362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.28.251.194:44113)
0808 03:58:15.570565 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:15.570570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:15.570591 (+ 21us) server_negotiation.cc:407] Connection header received
0808 03:58:15.570731 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:15.570736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:15.570806 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:15.570923 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:15.572335 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.573119 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.574407 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.574803 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.574901 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:15.577622 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:15.577642 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:15.577661 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:15.577727 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:15.580628 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:15.581410 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:15.581419 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:15.581422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:15.581578 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:15.581897 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:15.581900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:15.581902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:15.582289 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:15.582524 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:15.582779 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 03:58:15.582969 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 03:58:15.583990 9246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:15.680823 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 9035
I20260808 03:58:15.710269 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 9278
I20260808 03:58:15.725092 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.711135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40952 (local address 127.28.251.195:34591)
0808 03:58:15.711347 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:15.711351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:15.711373 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:58:15.711532 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:15.711537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:15.711608 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:15.711694 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:15.713374 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.714137 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.715674 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:15.716162 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:15.716272 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:15.718858 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:15.718876 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:15.718893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:15.718938 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:15.722196 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:15.722932 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:15.722939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:15.722942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:15.723093 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:15.723563 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:15.723567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:15.723569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:15.723984 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:15.724219 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:15.724557 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 03:58:15.724739 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 03:58:15.725766 9490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:15.850875 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 9278
I20260808 03:58:15.879439 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 8202
I20260808 03:58:15.880657 8469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:15.992537 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:15.991760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51200)
0808 03:58:15.992243 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:15.992341 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":246,"threads_started":1}
I20260808 03:58:16.001542 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:16.001100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40631 (local address 127.0.0.1:51206)
0808 03:58:16.001281 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:16.001376 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 03:58:16.007527 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 8202
I20260808 03:58:16.040953 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 8361
I20260808 03:58:16.042309 8763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:16.170279 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 8361
2026-08-08T03:58:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10561 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:58:16 dist-test-slave-pww0 krb5kdc[9714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:58:16 dist-test-slave-pww0 krb5kdc[9714](info): set up 2 sockets
Aug 08 03:58:16 dist-test-slave-pww0 krb5kdc[9714](info): commencing operation
krb5kdc: starting...
W20260808 03:58:17.456477 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.007s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:58:17 dist-test-slave-pww0 krb5kdc[9714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:58:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:58:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.254.keytab.
Entry for principal oryx/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.254.keytab.
I20260808 03:58:17.749334 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43799
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.254.keytab
--principal=oryx/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:43799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:18.106966 9737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:18.107339 9737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:18.107509 9737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:18.115144 9737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:58:18.115311 9737 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 03:58:18.115398 9737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:18.115520 9737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:18.115589 9737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:58:18.115677 9737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:58:18.126205 9737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:43799
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.254.keytab
--principal=oryx/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:18.127782 9737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:18.129293 9737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:18.141588 9746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:18.141549 9750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:18.141614 9748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:18.142576 9737 server_base.cc:1034] running on GCE node
I20260808 03:58:18.143496 9737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:18.145099 9737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:18.146337 9737 hybrid_clock.cc:648] HybridClock initialized: now 1786161498146252 us; error 53 us; skew 500 ppm
I20260808 03:58:18.146693 9737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:18 dist-test-slave-pww0 krb5kdc[9714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161498, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:18.149950 9737 init.cc:377] Logged in from keytab as oryx/127.28.251.254@KRBTEST.COM (short username oryx)
I20260808 03:58:18.151592 9737 webserver.cc:460] Webserver started at http://127.28.251.254:37677/ using document root <none> and password file <none>
I20260808 03:58:18.152365 9737 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:18.152479 9737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:18.152808 9737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:18.156137 9737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "7623b54c346448dfacb85b3389d119f7"
format_stamp: "Formatted at 2026-08-08 03:58:18 on dist-test-slave-pww0"
server_key: "6a75e3180e86fc5ab63de82555e70e0d"
server_key_iv: "22a14720cff059f920f70b50d87a76a3"
server_key_version: "encryptionkey@0"
I20260808 03:58:18.157006 9737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "7623b54c346448dfacb85b3389d119f7"
format_stamp: "Formatted at 2026-08-08 03:58:18 on dist-test-slave-pww0"
server_key: "6a75e3180e86fc5ab63de82555e70e0d"
server_key_iv: "22a14720cff059f920f70b50d87a76a3"
server_key_version: "encryptionkey@0"
I20260808 03:58:18.162685 9737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 03:58:18.166702 9762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:18.168241 9737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:18.168440 9737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "7623b54c346448dfacb85b3389d119f7"
format_stamp: "Formatted at 2026-08-08 03:58:18 on dist-test-slave-pww0"
server_key: "6a75e3180e86fc5ab63de82555e70e0d"
server_key_iv: "22a14720cff059f920f70b50d87a76a3"
server_key_version: "encryptionkey@0"
I20260808 03:58:18.168668 9737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:18.185568 9737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:18.190148 9737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:18.190441 9737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:18.214565 9737 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:43799
I20260808 03:58:18.214574 9854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:43799 every 8 connection(s)
I20260808 03:58:18.216135 9737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:58:18.219677 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 9737
I20260808 03:58:18.219908 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:58:18.220602 29679 external_mini_cluster.cc:1428] Setting key 405fc93224acd6709c17c20f7fcd2427
I20260808 03:58:18.220598 9855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:18.236161 9855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7: Bootstrap starting.
Aug 08 03:58:18 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:18.240502 9855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:18.242012 9855 log.cc:824] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:18.246021 9855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7: No bootstrap required, opened a new log
I20260808 03:58:18.247211 9860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:18.222639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.28.251.254:43799)
0808 03:58:18.223411 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:18.223424 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:18.223506 (+ 82us) server_negotiation.cc:407] Connection header received
0808 03:58:18.225576 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:18.225602 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:18.226066 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:18.226451 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:18.228494 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:18.229872 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:18.231698 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:18.232466 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:18.232617 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:18.237666 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:18.237681 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:18.237707 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:18.237783 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:18.241816 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:18.242707 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:18.242717 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:18.242728 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:18.242931 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:18.243438 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:18.243444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:18.243447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:18.244392 (+ 945us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:18.244723 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:18.245205 (+ 482us) server_negotiation.cc:299] Negotiation successful
0808 03:58:18.245687 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":334,"threads_started":1}
I20260808 03:58:18.254774 9855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7623b54c346448dfacb85b3389d119f7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43799 } }
I20260808 03:58:18.255261 9855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:18.255389 9855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7623b54c346448dfacb85b3389d119f7, State: Initialized, Role: FOLLOWER
I20260808 03:58:18.256067 9855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7623b54c346448dfacb85b3389d119f7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43799 } }
I20260808 03:58:18.256317 9855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:58:18.256453 9855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:58:18.256618 9855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:18.259131 9855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7623b54c346448dfacb85b3389d119f7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43799 } }
I20260808 03:58:18.259765 9855 leader_election.cc:302] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7623b54c346448dfacb85b3389d119f7; no voters:
I20260808 03:58:18.260349 9855 leader_election.cc:288] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:58:18.260512 9864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:18.261194 9864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [term 1 LEADER]: Becoming Leader. State: Replica: 7623b54c346448dfacb85b3389d119f7, State: Running, Role: LEADER
I20260808 03:58:18.261765 9864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7623b54c346448dfacb85b3389d119f7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43799 } }
I20260808 03:58:18.262494 9855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:58:18.264931 9867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7623b54c346448dfacb85b3389d119f7. Latest consensus state: current_term: 1 leader_uuid: "7623b54c346448dfacb85b3389d119f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7623b54c346448dfacb85b3389d119f7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43799 } } }
I20260808 03:58:18.265360 9867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:18.267131 9865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7623b54c346448dfacb85b3389d119f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7623b54c346448dfacb85b3389d119f7" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43799 } } }
I20260808 03:58:18.267375 9865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:18.267976 9876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:58:18.273032 9876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:58:18.283279 9876 catalog_manager.cc:1347] Generated new cluster ID: a4f13f913d3b42b69b1138da5da1749e
I20260808 03:58:18.283409 9876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:58:18.295375 9876 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:58:18.297313 9876 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:58:18.305838 9876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7: Generated new TSK 0
I20260808 03:58:18.306789 9876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab.
Entry for principal oryx/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab.
I20260808 03:58:18.391053 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43799
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab
--principal=oryx/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:18.756500 9902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:18.756835 9902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:18.756940 9902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:18.764734 9902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:18.764870 9902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:18.764958 9902 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 03:58:18.765043 9902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:58:18.775906 9902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab
--principal=oryx/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.251.254:43799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:18.777266 9902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:18.778657 9902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:18.792582 9910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:18.792850 9912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:18.793036 9909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:18.794082 9902 server_base.cc:1034] running on GCE node
I20260808 03:58:18.794811 9902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:18.795943 9902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:18.797179 9902 hybrid_clock.cc:648] HybridClock initialized: now 1786161498797107 us; error 49 us; skew 500 ppm
I20260808 03:58:18.797536 9902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:18 dist-test-slave-pww0 krb5kdc[9714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161498, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:18.805092 9902 init.cc:377] Logged in from keytab as oryx/127.28.251.193@KRBTEST.COM (short username oryx)
I20260808 03:58:18.806828 9902 webserver.cc:460] Webserver started at http://127.28.251.193:45391/ using document root <none> and password file <none>
I20260808 03:58:18.807679 9902 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:18.807803 9902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:18.808168 9902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:18.811389 9902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "d42d86efb50b4863819800dc62467b6b"
format_stamp: "Formatted at 2026-08-08 03:58:18 on dist-test-slave-pww0"
server_key: "c61d93092bf63ab8de508bcc4ebae2be"
server_key_iv: "7dc52fa4d2e427055035ca798cb14771"
server_key_version: "encryptionkey@0"
I20260808 03:58:18.812239 9902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "d42d86efb50b4863819800dc62467b6b"
format_stamp: "Formatted at 2026-08-08 03:58:18 on dist-test-slave-pww0"
server_key: "c61d93092bf63ab8de508bcc4ebae2be"
server_key_iv: "7dc52fa4d2e427055035ca798cb14771"
server_key_version: "encryptionkey@0"
I20260808 03:58:18.817797 9902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 03:58:18.821712 9923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:18.823155 9902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 03:58:18.823333 9902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "d42d86efb50b4863819800dc62467b6b"
format_stamp: "Formatted at 2026-08-08 03:58:18 on dist-test-slave-pww0"
server_key: "c61d93092bf63ab8de508bcc4ebae2be"
server_key_iv: "7dc52fa4d2e427055035ca798cb14771"
server_key_version: "encryptionkey@0"
I20260808 03:58:18.823621 9902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:18.839941 9902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:18.844761 9902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:18.845115 9902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:18.848070 9902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:18.848161 9902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:18.848344 9902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:18.848428 9902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 03:58:18 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161498, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:18.873231 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:18.849018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:54271 (local address 127.28.251.254:43799)
0808 03:58:18.849526 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:18.849531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:18.850891 (+ 1360us) server_negotiation.cc:407] Connection header received
0808 03:58:18.853151 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:18.853157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:18.853270 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:18.853375 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:18.855880 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:18.856827 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:18.859424 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:18.860037 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:18.860174 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:18.865537 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:18.865545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:18.865549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:18.865601 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:18.869027 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:18.870203 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:18.870209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:18.870213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:18.870397 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:18.871107 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:18.871113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:18.871116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:18.871441 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:18.871623 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:18.872553 (+ 930us) server_negotiation.cc:299] Negotiation successful
0808 03:58:18.872821 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":268,"threads_started":1}
I20260808 03:58:18.874380 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:18.849744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43799 (local address 127.28.251.193:54271)
0808 03:58:18.850667 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:18.850714 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:18.852813 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:18.853712 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:18.853725 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:18.854604 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:18.855652 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:18.855678 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:18.857070 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:18.857074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:18.859239 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:18.859256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:18.860337 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:18.860346 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:18.860743 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:18.861706 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:18.861750 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:18.865186 (+ 3436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:18.869210 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:18.869237 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:18.869260 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:18.870046 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:18.870570 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:18.870579 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:18.870582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:18.870929 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:18.871629 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:18.871638 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:18.872093 (+ 455us) client_negotiation.cc:769] Sending connection context
0808 03:58:18.872397 (+ 304us) client_negotiation.cc:240] Negotiation successful
0808 03:58:18.873043 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":374,"threads_started":1}
I20260808 03:58:18.896302 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:18.880959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:41113 (local address 127.28.251.254:43799)
0808 03:58:18.881165 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:18.881172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:18.882015 (+ 843us) server_negotiation.cc:407] Connection header received
0808 03:58:18.882194 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:18.882198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:18.882272 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:18.882405 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:18.883992 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:18.884987 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:18.886677 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:18.887175 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:18.887325 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:18.889792 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:18.889798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:18.889801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:18.889844 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:18.892955 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:18.893974 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:18.893980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:18.893984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:18.894168 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:18.894622 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:18.894628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:18.894631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:18.895037 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:18.895197 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:18.895672 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 03:58:18.895927 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 03:58:18.896454 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:18.881013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43799 (local address 127.28.251.193:41113)
0808 03:58:18.881868 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:18.881899 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:18.882077 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:18.882447 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:18.882452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:18.883142 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:18.883705 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:18.883725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:18.885137 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:18.885143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:18.886510 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:18.886525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:18.887419 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:18.887427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:18.887670 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:18.888366 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:18.888393 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:18.889546 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:18.893171 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:18.893177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:18.893182 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:18.893818 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:18.894327 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:18.894332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:18.894336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:18.894499 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:18.895167 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:18.895173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:18.895377 (+ 204us) client_negotiation.cc:769] Sending connection context
0808 03:58:18.895559 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 03:58:18.895993 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":461,"threads_started":1}
I20260808 03:58:18.902454 9902 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:44275
I20260808 03:58:18.902491 10121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:44275 every 8 connection(s)
I20260808 03:58:18.903872 9902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:58:18.906289 10122 heartbeater.cc:347] Connected to a master server at 127.28.251.254:43799
I20260808 03:58:18.906562 10122 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:18.907287 10122 heartbeater.cc:511] Master 127.28.251.254:43799 requested a full tablet report, sending...
I20260808 03:58:18.909689 9788 ts_manager.cc:194] Registered new tserver with Master: d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275)
I20260808 03:58:18.911259 9788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:54271
I20260808 03:58:18.913039 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 9902
I20260808 03:58:18.913259 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:58:18.914156 29679 external_mini_cluster.cc:1428] Setting key ec37b92301dc1092f47aa1e66490c894
WARNING: no policy specified for oryx/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab.
Entry for principal oryx/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab.
I20260808 03:58:18.979116 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43799
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab
--principal=oryx/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:19.348220 10126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:19.348560 10126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:19.348706 10126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:19.356407 10126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:19.356570 10126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:19.356716 10126 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 03:58:19.356837 10126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:58:19.367274 10126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab
--principal=oryx/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.251.254:43799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:19.368719 10126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:19.370136 10126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:19.384171 10132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:19.384224 10139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:19.384246 10133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:19.385483 10126 server_base.cc:1034] running on GCE node
I20260808 03:58:19.386171 10126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:19.387212 10126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:19.388464 10126 hybrid_clock.cc:648] HybridClock initialized: now 1786161499388373 us; error 66 us; skew 500 ppm
I20260808 03:58:19.388811 10126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:19 dist-test-slave-pww0 krb5kdc[9714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161499, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:19.392352 10126 init.cc:377] Logged in from keytab as oryx/127.28.251.194@KRBTEST.COM (short username oryx)
I20260808 03:58:19.393976 10126 webserver.cc:460] Webserver started at http://127.28.251.194:36451/ using document root <none> and password file <none>
I20260808 03:58:19.394713 10126 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:19.394809 10126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:19.395133 10126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:19.398602 10126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "50de18ad8be445029bbadc1679b631b4"
format_stamp: "Formatted at 2026-08-08 03:58:19 on dist-test-slave-pww0"
server_key: "907c94f579aacbe93b874f1084207062"
server_key_iv: "7f6b381bb07ccec19cddbbff4ec871ad"
server_key_version: "encryptionkey@0"
I20260808 03:58:19.399542 10126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "50de18ad8be445029bbadc1679b631b4"
format_stamp: "Formatted at 2026-08-08 03:58:19 on dist-test-slave-pww0"
server_key: "907c94f579aacbe93b874f1084207062"
server_key_iv: "7f6b381bb07ccec19cddbbff4ec871ad"
server_key_version: "encryptionkey@0"
I20260808 03:58:19.405510 10126 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 03:58:19.409664 10150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:19.411221 10126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 03:58:19.411423 10126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "50de18ad8be445029bbadc1679b631b4"
format_stamp: "Formatted at 2026-08-08 03:58:19 on dist-test-slave-pww0"
server_key: "907c94f579aacbe93b874f1084207062"
server_key_iv: "7f6b381bb07ccec19cddbbff4ec871ad"
server_key_version: "encryptionkey@0"
I20260808 03:58:19.411715 10126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:19.430609 10126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:19.435439 10126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:19.435791 10126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:19.438691 10126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:19.438794 10126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:19.438935 10126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:19.439049 10126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 08 03:58:19 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161499, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:19.465102 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:19.439385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:39449 (local address 127.28.251.254:43799)
0808 03:58:19.440003 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:19.440010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:19.441297 (+ 1287us) server_negotiation.cc:407] Connection header received
0808 03:58:19.444445 (+ 3148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:19.444452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:19.444553 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:19.444673 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:19.447217 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:19.448243 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:19.450748 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:19.451509 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:19.451672 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:19.457160 (+ 5488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:19.457167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:19.457173 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:19.457233 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:19.460639 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:19.461735 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:19.461741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:19.461745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:19.461902 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:19.462604 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:19.462609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:19.462612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:19.463022 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:19.463175 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:19.464340 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0808 03:58:19.464645 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":369,"threads_started":1}
I20260808 03:58:19.465952 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:19.440101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43799 (local address 127.28.251.194:39449)
0808 03:58:19.441077 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:19.441134 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:19.443881 (+ 2747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:19.444986 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:19.445021 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:19.445922 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:19.446981 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:19.447007 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:19.448407 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:19.448412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:19.450533 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:19.450560 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:19.451806 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:19.451813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:19.452058 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:19.453014 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:19.453042 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:19.456783 (+ 3741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:19.460826 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:19.460841 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:19.460856 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:19.461573 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:19.462046 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:19.462053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:19.462056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:19.462469 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:19.463201 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:19.463211 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:19.463823 (+ 612us) client_negotiation.cc:769] Sending connection context
0808 03:58:19.464170 (+ 347us) client_negotiation.cc:240] Negotiation successful
0808 03:58:19.464697 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":400,"threads_started":1}
I20260808 03:58:19.487733 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:19.472727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:53967 (local address 127.28.251.254:43799)
0808 03:58:19.472962 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:19.472969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:19.473481 (+ 512us) server_negotiation.cc:407] Connection header received
0808 03:58:19.473655 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:19.473661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:19.473757 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:19.473889 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:19.474810 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:19.476037 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:19.478158 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:19.478776 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:19.478923 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:19.481412 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:19.481418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:19.481423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:19.481469 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:19.484789 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:19.485666 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:19.485672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:19.485676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:19.485818 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:19.486185 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:19.486191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:19.486194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:19.486544 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:19.486701 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:19.487128 (+ 427us) server_negotiation.cc:299] Negotiation successful
0808 03:58:19.487350 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260808 03:58:19.487905 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:19.472675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43799 (local address 127.28.251.194:53967)
0808 03:58:19.473373 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:19.473390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:19.473520 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:19.473895 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:19.473899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:19.474352 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:19.474662 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:19.474672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:19.476220 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:19.476226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:19.477917 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:19.477938 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:19.478983 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:19.478990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:19.479127 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:19.479884 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:19.479903 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:19.481179 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:19.484914 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:19.484920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:19.484924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:19.485534 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:19.485938 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:19.485943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:19.485947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:19.486080 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:19.486665 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:19.486670 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:19.486880 (+ 210us) client_negotiation.cc:769] Sending connection context
0808 03:58:19.487003 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 03:58:19.487377 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":357,"threads_started":1}
I20260808 03:58:19.489888 10126 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:32979
I20260808 03:58:19.489914 10355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:32979 every 8 connection(s)
I20260808 03:58:19.491439 10126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:58:19.493839 10356 heartbeater.cc:347] Connected to a master server at 127.28.251.254:43799
I20260808 03:58:19.494059 10356 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:19.494724 10356 heartbeater.cc:511] Master 127.28.251.254:43799 requested a full tablet report, sending...
I20260808 03:58:19.496107 9788 ts_manager.cc:194] Registered new tserver with Master: 50de18ad8be445029bbadc1679b631b4 (127.28.251.194:32979)
I20260808 03:58:19.496948 9788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:39449
I20260808 03:58:19.500445 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 10126
I20260808 03:58:19.500609 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:58:19.501055 29679 external_mini_cluster.cc:1428] Setting key ba56bedf5380e1c311ad653aae0a5a48
WARNING: no policy specified for oryx/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab.
Entry for principal oryx/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab.
I20260808 03:58:19.562685 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43799
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab
--principal=oryx/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 03:58:19.915145 10122 heartbeater.cc:503] Master 127.28.251.254:43799 was elected leader, sending a full tablet report...
W20260808 03:58:19.927794 10366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:19.928131 10366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:19.928288 10366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:19.935982 10366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:19.936123 10366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:19.936259 10366 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 03:58:19.936393 10366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:58:19.946941 10366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab
--principal=oryx/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.251.254:43799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:19.948458 10366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:19.949954 10366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:19.964407 10375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:19.964375 10378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:19.964689 10374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:19.965402 10366 server_base.cc:1034] running on GCE node
I20260808 03:58:19.966092 10366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:19.967142 10366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:19.968394 10366 hybrid_clock.cc:648] HybridClock initialized: now 1786161499968297 us; error 67 us; skew 500 ppm
I20260808 03:58:19.968757 10366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:19 dist-test-slave-pww0 krb5kdc[9714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161499, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:19.972256 10366 init.cc:377] Logged in from keytab as oryx/127.28.251.195@KRBTEST.COM (short username oryx)
I20260808 03:58:19.974076 10366 webserver.cc:460] Webserver started at http://127.28.251.195:43717/ using document root <none> and password file <none>
I20260808 03:58:19.974848 10366 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:19.974965 10366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:19.975303 10366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:19.978645 10366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "bc3134c83cc74c2bb6e2aa70e96af964"
format_stamp: "Formatted at 2026-08-08 03:58:19 on dist-test-slave-pww0"
server_key: "583ba258f63eb92349a1dfaa96fe9e4d"
server_key_iv: "9cd3e635ec2266f7e4e1eb9499561337"
server_key_version: "encryptionkey@0"
I20260808 03:58:19.979575 10366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "bc3134c83cc74c2bb6e2aa70e96af964"
format_stamp: "Formatted at 2026-08-08 03:58:19 on dist-test-slave-pww0"
server_key: "583ba258f63eb92349a1dfaa96fe9e4d"
server_key_iv: "9cd3e635ec2266f7e4e1eb9499561337"
server_key_version: "encryptionkey@0"
I20260808 03:58:19.985332 10366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 03:58:19.989449 10392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:19.990960 10366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 03:58:19.991146 10366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "bc3134c83cc74c2bb6e2aa70e96af964"
format_stamp: "Formatted at 2026-08-08 03:58:19 on dist-test-slave-pww0"
server_key: "583ba258f63eb92349a1dfaa96fe9e4d"
server_key_iv: "9cd3e635ec2266f7e4e1eb9499561337"
server_key_version: "encryptionkey@0"
I20260808 03:58:19.991384 10366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:20.010367 10366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:20.014945 10366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:20.015246 10366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:20.018687 10366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:20.018795 10366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:20.018941 10366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:20.019001 10366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 03:58:20 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161499, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:20.046233 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.019409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:40919 (local address 127.28.251.254:43799)
0808 03:58:20.020039 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.020046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.021367 (+ 1321us) server_negotiation.cc:407] Connection header received
0808 03:58:20.024459 (+ 3092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.024465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.024574 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.024719 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:20.027839 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.028940 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.032114 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.032625 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.032740 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.038212 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:20.038221 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:20.038228 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:20.038285 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:20.042114 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:20.043426 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:20.043432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:20.043434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:20.043642 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:20.044201 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:20.044207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:20.044210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:20.044537 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:20.044700 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.045555 (+ 855us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.045818 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":307,"threads_started":1}
I20260808 03:58:20.047618 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.020144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43799 (local address 127.28.251.195:40919)
0808 03:58:20.021136 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:20.021205 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:20.024014 (+ 2809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:20.025145 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:20.025160 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:20.026194 (+ 1034us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:20.027451 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.027512 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.029169 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.029176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.031894 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.031918 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.032907 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.032916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.033380 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.034339 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:20.034388 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:20.037841 (+ 3453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:20.042338 (+ 4497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:20.042350 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:20.042386 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:20.043190 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:20.043777 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:20.043783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:20.043787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:20.044056 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:20.044689 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:20.044699 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:20.045147 (+ 448us) client_negotiation.cc:769] Sending connection context
0808 03:58:20.045421 (+ 274us) client_negotiation.cc:240] Negotiation successful
0808 03:58:20.046051 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":380,"threads_started":1}
I20260808 03:58:20.070271 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.055947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:44897 (local address 127.28.251.254:43799)
0808 03:58:20.056126 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.056130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.056797 (+ 667us) server_negotiation.cc:407] Connection header received
0808 03:58:20.056998 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.057003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.057091 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.057177 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:20.058192 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.058937 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.060689 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.061239 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.061365 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.063826 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:20.063833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:20.063838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:20.063885 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:20.067258 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:20.068161 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:20.068165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:20.068167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:20.068354 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:20.068758 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:20.068762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:20.068764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:20.069044 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:20.069205 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.069643 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.069882 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 03:58:20.070457 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.056050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43799 (local address 127.28.251.195:44897)
0808 03:58:20.056674 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:20.056692 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:20.056847 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:20.057227 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:20.057233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:20.057727 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:20.058033 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.058043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.059090 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.059095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.060475 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.060491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.061654 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.061658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.061802 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.062482 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:20.062501 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:20.063590 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:20.067519 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:20.067525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:20.067533 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:20.067993 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:20.068477 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:20.068482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:20.068486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:20.068630 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:20.069170 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:20.069175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:20.069380 (+ 205us) client_negotiation.cc:769] Sending connection context
0808 03:58:20.069505 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 03:58:20.069957 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":333,"threads_started":1}
I20260808 03:58:20.077090 10366 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:45145
I20260808 03:58:20.077131 10587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:45145 every 8 connection(s)
I20260808 03:58:20.078416 10366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:58:20.080600 10588 heartbeater.cc:347] Connected to a master server at 127.28.251.254:43799
I20260808 03:58:20.080853 10588 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:20.081558 10588 heartbeater.cc:511] Master 127.28.251.254:43799 requested a full tablet report, sending...
I20260808 03:58:20.083081 9788 ts_manager.cc:194] Registered new tserver with Master: bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195:45145)
I20260808 03:58:20.083760 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 10366
I20260808 03:58:20.083928 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:58:20.084055 9788 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:40919
I20260808 03:58:20.084475 29679 external_mini_cluster.cc:1428] Setting key 72118872dc149309638bf580bcd4b467
I20260808 03:58:20.087788 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 03:58:20 dist-test-slave-pww0 krb5kdc[9714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 03:58:20 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:20 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 03:58:20.155308 10406 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:20.143745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.28.251.254:43799)
0808 03:58:20.143996 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.144002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.144258 (+ 256us) server_negotiation.cc:407] Connection header received
0808 03:58:20.144492 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.144496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.144567 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.144659 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:20.146137 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.146901 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.148355 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.148783 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.148918 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.154970 (+ 6052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39034: BlockingRecv error: recv got EOF from 127.0.0.1:39034 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 08 03:58:20 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:20.173895 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.159220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.28.251.254:43799)
0808 03:58:20.159437 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.159444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.159750 (+ 306us) server_negotiation.cc:407] Connection header received
0808 03:58:20.159938 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.159944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.160035 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.160123 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:20.161378 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.162115 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.163498 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.164046 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.164167 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.168254 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:20.168262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:20.168267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:20.168318 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:20.170950 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:20.171827 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:20.171831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:20.171834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:20.171978 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:20.172399 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:20.172404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:20.172407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:20.172737 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:20.172981 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.173339 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.173586 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 03:58:20.178360 9788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:58:20.185659 9788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:58:20.230677 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.200016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44275 (local address 127.0.0.1:51298)
0808 03:58:20.215244 (+ 15228us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:20.215268 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:20.215434 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:20.220768 (+ 5334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:20.220778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:20.220890 (+ 112us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:20.221446 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.221462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.224700 (+ 3238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.224706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.227448 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.227496 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.229640 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.229646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.229867 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.229937 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 03:58:20.230051 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 03:58:20.230259 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15050,"thread_start_us":15530,"threads_started":1}
I20260808 03:58:20.232426 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.200364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.28.251.193:44275)
0808 03:58:20.219654 (+ 19290us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.219664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.219704 (+ 40us) server_negotiation.cc:407] Connection header received
0808 03:58:20.219851 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.219862 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.220185 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.221723 (+ 1538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:20.221985 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.223912 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.227732 (+ 3820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.229371 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.231623 (+ 2252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.231838 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.231920 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.232106 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19069,"thread_start_us":20023,"threads_started":1}
I20260808 03:58:20.234372 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.199318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:45145 (local address 127.0.0.1:43422)
0808 03:58:20.213909 (+ 14591us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:20.213964 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:20.214204 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:20.225757 (+ 11553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:20.225764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:20.225798 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:20.226262 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.226279 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.228665 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.228671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.230916 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.230932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.233703 (+ 2771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.233710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.233908 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.233948 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 03:58:20.234009 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 03:58:20.234113 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14251,"thread_start_us":15111,"threads_started":1}
I20260808 03:58:20.235538 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.199618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.28.251.195:45145)
0808 03:58:20.224659 (+ 25041us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.224667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.224718 (+ 51us) server_negotiation.cc:407] Connection header received
0808 03:58:20.224838 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.224847 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.225145 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.225388 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:20.226526 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.228497 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.231159 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.232678 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.234699 (+ 2021us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.234903 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.234991 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.235165 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24860,"thread_start_us":24675,"threads_started":1}
I20260808 03:58:20.236691 10002 tablet_service.cc:1467] Processing CreateTablet for tablet b9f8490064674e20a99097b206de9ecb (DEFAULT_TABLE table=test-table [id=bc27cde707dc4779bd66ac8c41698d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:20.238417 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f8490064674e20a99097b206de9ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:20.239203 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.215625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.28.251.194:32979)
0808 03:58:20.216292 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.216302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.226074 (+ 9772us) server_negotiation.cc:407] Connection header received
0808 03:58:20.227786 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.227796 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.228104 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.228292 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:20.230926 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.232487 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.236619 (+ 4132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.238021 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.238228 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.238416 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.238714 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.238903 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":3046,"threads_started":1}
I20260808 03:58:20.238952 10486 tablet_service.cc:1467] Processing CreateTablet for tablet b9f8490064674e20a99097b206de9ecb (DEFAULT_TABLE table=test-table [id=bc27cde707dc4779bd66ac8c41698d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:20.239938 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.219062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:32979 (local address 127.0.0.1:33306)
0808 03:58:20.225928 (+ 6866us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:20.225953 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:20.227558 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:20.228690 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:20.228695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:20.228722 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:20.229223 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.229235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.232681 (+ 3446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.232686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.236415 (+ 3729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.236433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.238282 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.238290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.238460 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.238494 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 03:58:20.239417 (+ 923us) client_negotiation.cc:240] Negotiation successful
0808 03:58:20.239598 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6691,"thread_start_us":7037,"threads_started":1}
I20260808 03:58:20.240574 10486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f8490064674e20a99097b206de9ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:20.243990 10235 tablet_service.cc:1467] Processing CreateTablet for tablet b9f8490064674e20a99097b206de9ecb (DEFAULT_TABLE table=test-table [id=bc27cde707dc4779bd66ac8c41698d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:20.246163 10235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f8490064674e20a99097b206de9ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:20.253034 10626 tablet_bootstrap.cc:492] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: Bootstrap starting.
I20260808 03:58:20.255232 10627 tablet_bootstrap.cc:492] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: Bootstrap starting.
I20260808 03:58:20.257258 10626 tablet_bootstrap.cc:654] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:20.258769 10626 log.cc:824] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:20.259080 10627 tablet_bootstrap.cc:654] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:20.260502 10627 log.cc:824] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:20.262378 10626 tablet_bootstrap.cc:492] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: No bootstrap required, opened a new log
I20260808 03:58:20.262384 10634 tablet_bootstrap.cc:492] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: Bootstrap starting.
I20260808 03:58:20.262784 10626 ts_tablet_manager.cc:1397] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260808 03:58:20.263975 10627 tablet_bootstrap.cc:492] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: No bootstrap required, opened a new log
I20260808 03:58:20.264281 10627 ts_tablet_manager.cc:1397] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 03:58:20.265648 10634 tablet_bootstrap.cc:654] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:20.266741 10634 log.cc:824] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:20.269956 10634 tablet_bootstrap.cc:492] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: No bootstrap required, opened a new log
I20260808 03:58:20.270234 10634 ts_tablet_manager.cc:1397] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260808 03:58:20.275071 10627 raft_consensus.cc:348] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } }
I20260808 03:58:20.275349 10626 raft_consensus.cc:348] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } }
I20260808 03:58:20.275712 10627 raft_consensus.cc:374] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:20.275820 10627 raft_consensus.cc:729] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42d86efb50b4863819800dc62467b6b, State: Initialized, Role: FOLLOWER
I20260808 03:58:20.275910 10626 raft_consensus.cc:374] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:20.276089 10626 raft_consensus.cc:729] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3134c83cc74c2bb6e2aa70e96af964, State: Initialized, Role: FOLLOWER
I20260808 03:58:20.276607 10627 consensus_queue.cc:260] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } }
I20260808 03:58:20.276793 10626 consensus_queue.cc:260] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } }
I20260808 03:58:20.278051 10588 heartbeater.cc:503] Master 127.28.251.254:43799 was elected leader, sending a full tablet report...
I20260808 03:58:20.278419 10634 raft_consensus.cc:348] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } }
I20260808 03:58:20.278918 10634 raft_consensus.cc:374] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:20.279006 10634 raft_consensus.cc:729] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50de18ad8be445029bbadc1679b631b4, State: Initialized, Role: FOLLOWER
I20260808 03:58:20.279621 10634 consensus_queue.cc:260] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } }
I20260808 03:58:20.280215 10626 ts_tablet_manager.cc:1428] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20260808 03:58:20.281823 10627 ts_tablet_manager.cc:1428] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260808 03:58:20.286857 10356 heartbeater.cc:503] Master 127.28.251.254:43799 was elected leader, sending a full tablet report...
I20260808 03:58:20.287778 10634 ts_tablet_manager.cc:1428] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: Time spent starting tablet: real 0.017s user 0.015s sys 0.003s
W20260808 03:58:20.330158 10589 tablet.cc:2369] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 03:58:20.406711 10123 tablet.cc:2369] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 03:58:20.494052 10357 tablet.cc:2369] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:58:20.499449 10640 raft_consensus.cc:482] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:58:20.499837 10640 raft_consensus.cc:504] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } }
I20260808 03:58:20.501667 10640 leader_election.cc:288] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195:45145), d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275)
I20260808 03:58:20.511510 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.502241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:34737 (local address 127.28.251.193:44275)
0808 03:58:20.502477 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.502485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.503061 (+ 576us) server_negotiation.cc:407] Connection header received
0808 03:58:20.503245 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.503250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.503346 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.503435 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:20.504419 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.505533 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.508942 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.510209 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.510418 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.510502 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.511051 (+ 549us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.511212 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 03:58:20.511508 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.502241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:42393 (local address 127.28.251.195:45145)
0808 03:58:20.502447 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.502455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.503010 (+ 555us) server_negotiation.cc:407] Connection header received
0808 03:58:20.503226 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.503232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.503319 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.503435 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:20.504428 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.505533 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.508938 (+ 3405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.510208 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.510418 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.510502 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.511051 (+ 549us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.511230 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 03:58:20.511783 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.502008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44275 (local address 127.28.251.194:34737)
0808 03:58:20.502865 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:20.502890 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:20.503107 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:20.503534 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:20.503541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:20.503584 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:20.504199 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.504217 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.505729 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.505733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.508699 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.508722 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.510546 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.510551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.510798 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.510859 (+ 61us) client_negotiation.cc:769] Sending connection context
0808 03:58:20.510974 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 03:58:20.511196 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":316,"spinlock_wait_cycles":40064,"thread_start_us":479,"threads_started":1}
I20260808 03:58:20.511806 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.502137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:45145 (local address 127.28.251.194:42393)
0808 03:58:20.502893 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:20.502911 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:20.503061 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:20.503534 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:20.503541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:20.503593 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:20.504205 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.504217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.505727 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.505733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.508699 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.508722 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.510526 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.510534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.510805 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.510859 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 03:58:20.510974 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 03:58:20.511197 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":216,"spinlock_wait_cycles":6400,"thread_start_us":487,"threads_started":1}
I20260808 03:58:20.512916 10516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f8490064674e20a99097b206de9ecb" candidate_uuid: "50de18ad8be445029bbadc1679b631b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" is_pre_election: true
I20260808 03:58:20.512948 10038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f8490064674e20a99097b206de9ecb" candidate_uuid: "50de18ad8be445029bbadc1679b631b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42d86efb50b4863819800dc62467b6b" is_pre_election: true
I20260808 03:58:20.513441 10516 raft_consensus.cc:2454] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50de18ad8be445029bbadc1679b631b4 in term 0.
I20260808 03:58:20.513461 10038 raft_consensus.cc:2454] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50de18ad8be445029bbadc1679b631b4 in term 0.
I20260808 03:58:20.514089 10152 leader_election.cc:302] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50de18ad8be445029bbadc1679b631b4, d42d86efb50b4863819800dc62467b6b; no voters:
I20260808 03:58:20.514503 10640 raft_consensus.cc:2781] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:58:20.514643 10640 raft_consensus.cc:482] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:58:20.514755 10640 raft_consensus.cc:3037] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:20.517302 10640 raft_consensus.cc:504] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } }
I20260808 03:58:20.517967 10640 leader_election.cc:288] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [CANDIDATE]: Term 1 election: Requested vote from peers bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195:45145), d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275)
I20260808 03:58:20.518631 10038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f8490064674e20a99097b206de9ecb" candidate_uuid: "50de18ad8be445029bbadc1679b631b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42d86efb50b4863819800dc62467b6b"
I20260808 03:58:20.518857 10038 raft_consensus.cc:3037] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:20.519115 10516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f8490064674e20a99097b206de9ecb" candidate_uuid: "50de18ad8be445029bbadc1679b631b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3134c83cc74c2bb6e2aa70e96af964"
I20260808 03:58:20.519372 10516 raft_consensus.cc:3037] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:20.522380 10038 raft_consensus.cc:2454] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50de18ad8be445029bbadc1679b631b4 in term 1.
I20260808 03:58:20.522568 10516 raft_consensus.cc:2454] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50de18ad8be445029bbadc1679b631b4 in term 1.
I20260808 03:58:20.523074 10156 leader_election.cc:302] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50de18ad8be445029bbadc1679b631b4, bc3134c83cc74c2bb6e2aa70e96af964, d42d86efb50b4863819800dc62467b6b; no voters:
I20260808 03:58:20.523492 10640 raft_consensus.cc:2781] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:20.524012 10640 raft_consensus.cc:686] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 1 LEADER]: Becoming Leader. State: Replica: 50de18ad8be445029bbadc1679b631b4, State: Running, Role: LEADER
I20260808 03:58:20.524577 10640 consensus_queue.cc:237] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } }
I20260808 03:58:20.529634 9785 catalog_manager.cc:5495] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50de18ad8be445029bbadc1679b631b4 (127.28.251.194). New cstate: current_term: 1 leader_uuid: "50de18ad8be445029bbadc1679b631b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } health_report { overall_health: UNKNOWN } } }
I20260808 03:58:20.613724 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.608075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.28.251.194:32979)
0808 03:58:20.608288 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.608294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.608378 (+ 84us) server_negotiation.cc:407] Connection header received
0808 03:58:20.608641 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.608646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.608736 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.608878 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:58:20.609291 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.610040 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.611421 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.611875 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.611990 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.612502 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:58:20.613077 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:58:20.613222 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.613348 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.613444 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 03:58:20.622402 10516 raft_consensus.cc:1261] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Refusing update from remote peer 50de18ad8be445029bbadc1679b631b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:20.622406 10038 raft_consensus.cc:1261] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Refusing update from remote peer 50de18ad8be445029bbadc1679b631b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:20.623450 10649 consensus_queue.cc:1035] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:20.623837 10640 consensus_queue.cc:1035] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:20.635676 10654 mvcc.cc:205] Tried to move back new op lower bound from 7316117506538999808 to 7316117506152701952. Current Snapshot: MvccSnapshot[applied={T|T < 7316117506538999808}]
I20260808 03:58:20.651656 9788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:20.652007 9788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:20.656380 9788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7: Sending DeleteTablet for 3 replicas of tablet b9f8490064674e20a99097b206de9ecb
I20260808 03:58:20.657873 10235 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f8490064674e20a99097b206de9ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:20 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:33306
I20260808 03:58:20.657827 10486 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f8490064674e20a99097b206de9ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:20 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:43422
I20260808 03:58:20.658908 10002 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f8490064674e20a99097b206de9ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:20 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:51298
I20260808 03:58:20.659519 10671 tablet_replica.cc:330] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: stopping tablet replica
I20260808 03:58:20.660017 10671 raft_consensus.cc:2229] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:58:20.660732 10671 raft_consensus.cc:2258] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:20.661847 10670 tablet_replica.cc:330] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: stopping tablet replica
I20260808 03:58:20.662371 10670 raft_consensus.cc:2229] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:20.662240 9788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 03:58:20.662894 10670 raft_consensus.cc:2258] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 03:58:20.663067 9788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:58:20.664942 10671 ts_tablet_manager.cc:1905] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:20.668253 10670 ts_tablet_manager.cc:1905] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:20.668334 10675 tablet_replica.cc:330] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: stopping tablet replica
I20260808 03:58:20.668812 10675 raft_consensus.cc:2229] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:20.672129 10675 raft_consensus.cc:2258] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:20.672384 10002 tablet_service.cc:1467] Processing CreateTablet for tablet 0512b137a859437ba29bc3f706d783af (DEFAULT_TABLE table=test-table [id=5079146abc3c4f4eb4c9fec322fec24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:20.672910 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0512b137a859437ba29bc3f706d783af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:20.672894 10486 tablet_service.cc:1467] Processing CreateTablet for tablet 0512b137a859437ba29bc3f706d783af (DEFAULT_TABLE table=test-table [id=5079146abc3c4f4eb4c9fec322fec24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:20.673363 10486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0512b137a859437ba29bc3f706d783af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:20.675238 10235 tablet_service.cc:1467] Processing CreateTablet for tablet 0512b137a859437ba29bc3f706d783af (DEFAULT_TABLE table=test-table [id=5079146abc3c4f4eb4c9fec322fec24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:20.675525 10671 ts_tablet_manager.cc:1918] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:20.675762 10671 log.cc:1196] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/b9f8490064674e20a99097b206de9ecb
I20260808 03:58:20.676072 10235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0512b137a859437ba29bc3f706d783af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:20.676410 10671 ts_tablet_manager.cc:1939] T b9f8490064674e20a99097b206de9ecb P 50de18ad8be445029bbadc1679b631b4: Deleting consensus metadata
I20260808 03:58:20.678725 9765 catalog_manager.cc:4841] TS 50de18ad8be445029bbadc1679b631b4 (127.28.251.194:32979): tablet b9f8490064674e20a99097b206de9ecb (table test-table [id=bc27cde707dc4779bd66ac8c41698d82]) successfully deleted
I20260808 03:58:20.682646 10626 tablet_bootstrap.cc:492] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964: Bootstrap starting.
I20260808 03:58:20.683216 10634 tablet_bootstrap.cc:492] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4: Bootstrap starting.
I20260808 03:58:20.684551 10626 tablet_bootstrap.cc:654] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:20.685182 10634 tablet_bootstrap.cc:654] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:20.685612 10675 ts_tablet_manager.cc:1905] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:20.686903 10627 tablet_bootstrap.cc:492] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b: Bootstrap starting.
I20260808 03:58:20.689203 10627 tablet_bootstrap.cc:654] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:20.694715 10675 ts_tablet_manager.cc:1918] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:20.694975 10675 log.cc:1196] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/b9f8490064674e20a99097b206de9ecb
I20260808 03:58:20.695678 10675 ts_tablet_manager.cc:1939] T b9f8490064674e20a99097b206de9ecb P d42d86efb50b4863819800dc62467b6b: Deleting consensus metadata
I20260808 03:58:20.696741 10670 ts_tablet_manager.cc:1918] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:20.696964 10670 log.cc:1196] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/b9f8490064674e20a99097b206de9ecb
I20260808 03:58:20.698114 9763 catalog_manager.cc:4841] TS d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275): tablet b9f8490064674e20a99097b206de9ecb (table test-table [id=bc27cde707dc4779bd66ac8c41698d82]) successfully deleted
I20260808 03:58:20.699934 10627 tablet_bootstrap.cc:492] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b: No bootstrap required, opened a new log
I20260808 03:58:20.700076 10670 ts_tablet_manager.cc:1939] T b9f8490064674e20a99097b206de9ecb P bc3134c83cc74c2bb6e2aa70e96af964: Deleting consensus metadata
I20260808 03:58:20.700192 10627 ts_tablet_manager.cc:1397] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260808 03:58:20.701292 10627 raft_consensus.cc:348] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.701686 10627 raft_consensus.cc:374] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:20.701838 10627 raft_consensus.cc:729] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42d86efb50b4863819800dc62467b6b, State: Initialized, Role: FOLLOWER
I20260808 03:58:20.702171 10626 tablet_bootstrap.cc:492] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964: No bootstrap required, opened a new log
I20260808 03:58:20.702344 10626 ts_tablet_manager.cc:1397] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.010s
I20260808 03:58:20.702297 10627 consensus_queue.cc:260] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.702744 9765 catalog_manager.cc:4841] TS bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195:45145): tablet b9f8490064674e20a99097b206de9ecb (table test-table [id=bc27cde707dc4779bd66ac8c41698d82]) successfully deleted
I20260808 03:58:20.703147 10627 ts_tablet_manager.cc:1428] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 03:58:20.704799 10634 tablet_bootstrap.cc:492] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4: No bootstrap required, opened a new log
I20260808 03:58:20.704991 10634 ts_tablet_manager.cc:1397] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.003s
I20260808 03:58:20.705896 10634 raft_consensus.cc:348] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.706159 10634 raft_consensus.cc:374] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:20.706278 10634 raft_consensus.cc:729] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50de18ad8be445029bbadc1679b631b4, State: Initialized, Role: FOLLOWER
I20260808 03:58:20.706423 10626 raft_consensus.cc:348] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.706674 10626 raft_consensus.cc:374] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:20.706611 10634 consensus_queue.cc:260] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.706768 10626 raft_consensus.cc:729] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3134c83cc74c2bb6e2aa70e96af964, State: Initialized, Role: FOLLOWER
I20260808 03:58:20.707952 10634 ts_tablet_manager.cc:1428] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 03:58:20.707093 10626 consensus_queue.cc:260] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.709714 10626 ts_tablet_manager.cc:1428] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260808 03:58:20.852640 10639 raft_consensus.cc:482] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:58:20.852883 10639 raft_consensus.cc:504] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.854565 10639 leader_election.cc:288] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275), 50de18ad8be445029bbadc1679b631b4 (127.28.251.194:32979)
I20260808 03:58:20.863076 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.855056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:38701 (local address 127.28.251.194:32979)
0808 03:58:20.855286 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.855293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.856261 (+ 968us) server_negotiation.cc:407] Connection header received
0808 03:58:20.856493 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.856498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.856597 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.856701 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:20.857482 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.858648 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.861159 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.862015 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.862172 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.862257 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.862603 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.862748 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 03:58:20.863145 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.854871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:32979 (local address 127.28.251.195:38701)
0808 03:58:20.856151 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:20.856168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:20.856286 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:20.856752 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:20.856757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:20.856795 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:20.857185 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.857201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.858805 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.858810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.860954 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.860965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.862213 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.862219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.862432 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.862470 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 03:58:20.862568 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 03:58:20.862713 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"mutex_wait_us":41,"spinlock_wait_cycles":1408,"thread_start_us":630,"threads_started":1}
I20260808 03:58:20.863951 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.855275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:39317 (local address 127.28.251.193:44275)
0808 03:58:20.855508 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.855515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.855995 (+ 480us) server_negotiation.cc:407] Connection header received
0808 03:58:20.856419 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.856426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.856514 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.856571 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:20.857551 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.858335 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.861110 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.862349 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.862565 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.862635 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.863542 (+ 907us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.863694 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 03:58:20.864012 10272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0512b137a859437ba29bc3f706d783af" candidate_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50de18ad8be445029bbadc1679b631b4" is_pre_election: true
I20260808 03:58:20.864499 10272 raft_consensus.cc:2454] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc3134c83cc74c2bb6e2aa70e96af964 in term 0.
I20260808 03:58:20.864992 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.854853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44275 (local address 127.28.251.195:39317)
0808 03:58:20.855850 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:20.855875 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:20.856201 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:20.856752 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:20.856757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:20.856801 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:20.857325 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.857337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.858493 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.858498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.860901 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.860919 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.862808 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.862814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.862997 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.863029 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 03:58:20.864447 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0808 03:58:20.864604 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":79,"spinlock_wait_cycles":3968,"thread_start_us":994,"threads_started":1}
I20260808 03:58:20.865162 10397 leader_election.cc:302] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50de18ad8be445029bbadc1679b631b4, bc3134c83cc74c2bb6e2aa70e96af964; no voters:
I20260808 03:58:20.865669 10038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0512b137a859437ba29bc3f706d783af" candidate_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42d86efb50b4863819800dc62467b6b" is_pre_election: true
I20260808 03:58:20.865902 10038 raft_consensus.cc:2454] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc3134c83cc74c2bb6e2aa70e96af964 in term 0.
I20260808 03:58:20.866200 10639 raft_consensus.cc:2781] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:58:20.866335 10639 raft_consensus.cc:482] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:58:20.866444 10639 raft_consensus.cc:3037] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:20.869159 10639 raft_consensus.cc:504] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.869920 10639 leader_election.cc:288] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [CANDIDATE]: Term 1 election: Requested vote from peers d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275), 50de18ad8be445029bbadc1679b631b4 (127.28.251.194:32979)
I20260808 03:58:20.870487 10272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0512b137a859437ba29bc3f706d783af" candidate_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50de18ad8be445029bbadc1679b631b4"
I20260808 03:58:20.870534 10038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0512b137a859437ba29bc3f706d783af" candidate_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42d86efb50b4863819800dc62467b6b"
I20260808 03:58:20.870718 10272 raft_consensus.cc:3037] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:20.870795 10038 raft_consensus.cc:3037] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:20.873639 10272 raft_consensus.cc:2454] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc3134c83cc74c2bb6e2aa70e96af964 in term 1.
I20260808 03:58:20.873908 10038 raft_consensus.cc:2454] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc3134c83cc74c2bb6e2aa70e96af964 in term 1.
I20260808 03:58:20.874187 10397 leader_election.cc:302] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50de18ad8be445029bbadc1679b631b4, bc3134c83cc74c2bb6e2aa70e96af964; no voters:
I20260808 03:58:20.874527 10639 raft_consensus.cc:2781] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:20.875065 10639 raft_consensus.cc:686] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 LEADER]: Becoming Leader. State: Replica: bc3134c83cc74c2bb6e2aa70e96af964, State: Running, Role: LEADER
I20260808 03:58:20.875535 10639 consensus_queue.cc:237] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.879426 9785 catalog_manager.cc:5495] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195). New cstate: current_term: 1 leader_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } health_report { overall_health: HEALTHY } } }
Aug 08 03:58:20 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161498, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.254@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:20.940933 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.922390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43799 (local address 127.0.0.1:39054)
0808 03:58:20.924983 (+ 2593us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:20.925008 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:20.925203 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:20.926089 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:20.926095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:20.927085 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:20.927419 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.927436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.928681 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.928687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.930255 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:20.930273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.931057 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.931070 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:20.931267 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.932117 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:20.932144 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:20.935093 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:20.938195 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:20.938211 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:20.938220 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:20.938661 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:20.939026 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:20.939031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:20.939035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:20.939224 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:20.939830 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:20.939839 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:20.940006 (+ 167us) client_negotiation.cc:769] Sending connection context
0808 03:58:20.940159 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 03:58:20.940456 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2388,"thread_start_us":1714,"threads_started":1}
I20260808 03:58:20.940966 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:20.922432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.28.251.254:43799)
0808 03:58:20.925746 (+ 3314us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:20.925751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:20.925772 (+ 21us) server_negotiation.cc:407] Connection header received
0808 03:58:20.925824 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:20.925827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:20.925888 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:20.925974 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:20.927621 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.928501 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.930437 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:20.930865 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:20.930959 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:20.935330 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:20.935335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:20.935339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:20.935374 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:20.938042 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:20.938790 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:20.938793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:20.938796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:20.938920 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:20.939337 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:20.939340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:20.939342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:20.939690 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:20.939843 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:20.940271 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 03:58:20.940608 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":3194,"thread_start_us":3002,"threads_started":1}
I20260808 03:58:20.947773 9785 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:39054:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 03:58:20.948941 9785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:58:20.956923 10002 tablet_service.cc:1467] Processing CreateTablet for tablet 1ade0ff896a74735b3cc33d4ebfc03f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43720bbf5eb34ddb8e0f14a05fa61b85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 03:58:20.957670 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ade0ff896a74735b3cc33d4ebfc03f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:20.957576 10235 tablet_service.cc:1467] Processing CreateTablet for tablet 1ade0ff896a74735b3cc33d4ebfc03f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43720bbf5eb34ddb8e0f14a05fa61b85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 03:58:20.957623 10486 tablet_service.cc:1467] Processing CreateTablet for tablet 1ade0ff896a74735b3cc33d4ebfc03f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43720bbf5eb34ddb8e0f14a05fa61b85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 03:58:20.958215 10235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ade0ff896a74735b3cc33d4ebfc03f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:20.958258 10486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ade0ff896a74735b3cc33d4ebfc03f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:20.966497 10627 tablet_bootstrap.cc:492] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: Bootstrap starting.
I20260808 03:58:20.966499 10634 tablet_bootstrap.cc:492] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4: Bootstrap starting.
I20260808 03:58:20.968436 10627 tablet_bootstrap.cc:654] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:20.968442 10634 tablet_bootstrap.cc:654] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:20.969640 10626 tablet_bootstrap.cc:492] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964: Bootstrap starting.
I20260808 03:58:20.971797 10626 tablet_bootstrap.cc:654] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:20.979058 10634 tablet_bootstrap.cc:492] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4: No bootstrap required, opened a new log
I20260808 03:58:20.979331 10634 ts_tablet_manager.cc:1397] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260808 03:58:20.980314 10634 raft_consensus.cc:348] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.980621 10634 raft_consensus.cc:374] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:20.980717 10634 raft_consensus.cc:729] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50de18ad8be445029bbadc1679b631b4, State: Initialized, Role: FOLLOWER
I20260808 03:58:20.981001 10634 consensus_queue.cc:260] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.981572 10663 tablet_replica.cc:439] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } }
I20260808 03:58:20.981936 10663 tablet_replica.cc:442] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:58:20.981993 10626 tablet_bootstrap.cc:492] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964: No bootstrap required, opened a new log
I20260808 03:58:20.982208 10626 ts_tablet_manager.cc:1397] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260808 03:58:20.983076 10634 ts_tablet_manager.cc:1428] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 03:58:20.983106 10626 raft_consensus.cc:348] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.983353 10626 raft_consensus.cc:374] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:20.983441 10626 raft_consensus.cc:729] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3134c83cc74c2bb6e2aa70e96af964, State: Initialized, Role: FOLLOWER
I20260808 03:58:20.983799 10626 consensus_queue.cc:260] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.984241 10639 tablet_replica.cc:439] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } }
I20260808 03:58:20.984555 10639 tablet_replica.cc:442] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:58:20.985513 10627 tablet_bootstrap.cc:492] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: No bootstrap required, opened a new log
I20260808 03:58:20.985698 10627 ts_tablet_manager.cc:1397] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.007s
I20260808 03:58:20.986642 10626 ts_tablet_manager.cc:1428] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260808 03:58:20.986545 10627 raft_consensus.cc:348] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.986791 10627 raft_consensus.cc:374] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:20.986863 10627 raft_consensus.cc:729] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42d86efb50b4863819800dc62467b6b, State: Initialized, Role: FOLLOWER
I20260808 03:58:20.987111 10627 consensus_queue.cc:260] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:20.987622 10637 tablet_replica.cc:439] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } }
I20260808 03:58:20.987977 10637 tablet_replica.cc:442] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:58:20.989158 10627 ts_tablet_manager.cc:1428] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 03:58:21.008571 10637 raft_consensus.cc:482] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:58:21.008801 10637 raft_consensus.cc:504] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.010517 10637 leader_election.cc:288] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50de18ad8be445029bbadc1679b631b4 (127.28.251.194:32979), bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195:45145)
I20260808 03:58:21.021171 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.011038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:46207 (local address 127.28.251.194:32979)
0808 03:58:21.011300 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:21.011308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:21.011836 (+ 528us) server_negotiation.cc:407] Connection header received
0808 03:58:21.012104 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:21.012111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:21.012194 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:21.012269 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:21.013468 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.014730 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.018026 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.019544 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.019745 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.019840 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:21.020566 (+ 726us) server_negotiation.cc:299] Negotiation successful
0808 03:58:21.020783 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 03:58:21.022173 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.010955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:32979 (local address 127.28.251.193:46207)
0808 03:58:21.011711 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:21.011730 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:21.011886 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:21.012773 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:21.012778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:21.012813 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:21.013224 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:21.013237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.014939 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.014945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:21.017765 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:21.017788 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.020096 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.020104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:21.020321 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.020372 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 03:58:21.021560 (+ 1188us) client_negotiation.cc:240] Negotiation successful
0808 03:58:21.021756 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":415,"threads_started":1}
I20260808 03:58:21.024009 10272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ade0ff896a74735b3cc33d4ebfc03f7" candidate_uuid: "d42d86efb50b4863819800dc62467b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50de18ad8be445029bbadc1679b631b4" is_pre_election: true
I20260808 03:58:21.024346 10272 raft_consensus.cc:2454] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42d86efb50b4863819800dc62467b6b in term 0.
I20260808 03:58:21.024941 9926 leader_election.cc:302] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50de18ad8be445029bbadc1679b631b4, d42d86efb50b4863819800dc62467b6b; no voters:
I20260808 03:58:21.025426 10637 raft_consensus.cc:2781] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:58:21.025545 10637 raft_consensus.cc:482] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:58:21.025645 10637 raft_consensus.cc:3037] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:21.028748 10637 raft_consensus.cc:504] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.029425 10637 leader_election.cc:288] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [CANDIDATE]: Term 1 election: Requested vote from peers 50de18ad8be445029bbadc1679b631b4 (127.28.251.194:32979), bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195:45145)
I20260808 03:58:21.029929 10272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ade0ff896a74735b3cc33d4ebfc03f7" candidate_uuid: "d42d86efb50b4863819800dc62467b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50de18ad8be445029bbadc1679b631b4"
I20260808 03:58:21.030165 10272 raft_consensus.cc:3037] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:21.030402 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.011924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:45145 (local address 127.28.251.193:51337)
0808 03:58:21.022491 (+ 10567us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:21.022514 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:21.022656 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:21.023556 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:21.023560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:21.023591 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:21.023756 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:21.023769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.025848 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.025854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:21.027752 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:21.027791 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.029541 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.029548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:21.029749 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.029789 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 03:58:21.029867 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 03:58:21.030021 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10440,"thread_start_us":11128,"threads_started":1}
I20260808 03:58:21.031369 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.012143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:51337 (local address 127.28.251.195:45145)
0808 03:58:21.023197 (+ 11054us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:21.023203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:21.023225 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:58:21.023299 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:21.023304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:21.023400 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:21.023541 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:21.023936 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.025685 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.027961 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.029314 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.030709 (+ 1395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.030798 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:21.030858 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 03:58:21.031008 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10922,"thread_start_us":10746,"threads_started":1}
I20260808 03:58:21.032145 10516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ade0ff896a74735b3cc33d4ebfc03f7" candidate_uuid: "d42d86efb50b4863819800dc62467b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" is_pre_election: true
I20260808 03:58:21.032431 10516 raft_consensus.cc:2454] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42d86efb50b4863819800dc62467b6b in term 0.
I20260808 03:58:21.032446 10515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ade0ff896a74735b3cc33d4ebfc03f7" candidate_uuid: "d42d86efb50b4863819800dc62467b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3134c83cc74c2bb6e2aa70e96af964"
I20260808 03:58:21.032884 10515 raft_consensus.cc:3037] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:21.033209 10272 raft_consensus.cc:2454] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42d86efb50b4863819800dc62467b6b in term 1.
I20260808 03:58:21.033707 9926 leader_election.cc:302] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50de18ad8be445029bbadc1679b631b4, d42d86efb50b4863819800dc62467b6b; no voters:
I20260808 03:58:21.034003 10637 raft_consensus.cc:2781] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:21.034502 10637 raft_consensus.cc:686] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [term 1 LEADER]: Becoming Leader. State: Replica: d42d86efb50b4863819800dc62467b6b, State: Running, Role: LEADER
I20260808 03:58:21.034961 10637 consensus_queue.cc:237] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.036124 10515 raft_consensus.cc:2454] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42d86efb50b4863819800dc62467b6b in term 1.
I20260808 03:58:21.037413 10711 tablet_replica.cc:439] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: TxnStatusTablet state changed. Reason: New leader d42d86efb50b4863819800dc62467b6b. Latest consensus state: current_term: 1 leader_uuid: "d42d86efb50b4863819800dc62467b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } }
I20260808 03:58:21.037932 10711 tablet_replica.cc:442] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: This TxnStatusTablet replica's current role is: LEADER
I20260808 03:58:21.039146 9788 catalog_manager.cc:5495] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b reported cstate change: term changed from 0 to 1, leader changed from <none> to d42d86efb50b4863819800dc62467b6b (127.28.251.193). New cstate: current_term: 1 leader_uuid: "d42d86efb50b4863819800dc62467b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } health_report { overall_health: UNKNOWN } } }
I20260808 03:58:21.039953 10715 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 03:58:21.040174 10715 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 03:58:21.070118 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.063961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44275 (local address 127.0.0.1:51312)
0808 03:58:21.064198 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:21.064220 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:21.064388 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:21.064955 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:21.064960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:21.064972 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 03:58:21.065156 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:21.065165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.066236 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.066242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:21.067275 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:21.067288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.068155 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.068161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:21.068382 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.068468 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 03:58:21.069521 (+ 1053us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 03:58:21.069558 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 03:58:21.069633 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 03:58:21.069810 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 03:58:21.070125 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.064087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.28.251.193:44275)
0808 03:58:21.064262 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:21.064269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:21.064293 (+ 24us) server_negotiation.cc:407] Connection header received
0808 03:58:21.064623 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:21.064628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:21.064770 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:21.064912 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:58:21.065291 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.066086 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.067441 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.068003 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.068125 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.068671 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:58:21.069400 (+ 729us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:58:21.069544 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:21.069666 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 03:58:21.069761 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 03:58:21.075088 10515 raft_consensus.cc:1261] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Refusing update from remote peer d42d86efb50b4863819800dc62467b6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:21.075578 10272 raft_consensus.cc:1261] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Refusing update from remote peer d42d86efb50b4863819800dc62467b6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:21.076658 10637 consensus_queue.cc:1035] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:21.077385 10637 consensus_queue.cc:1035] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:21.102041 10639 tablet_replica.cc:439] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964: TxnStatusTablet state changed. Reason: New leader d42d86efb50b4863819800dc62467b6b. Latest consensus state: current_term: 1 leader_uuid: "d42d86efb50b4863819800dc62467b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } }
I20260808 03:58:21.102442 10639 tablet_replica.cc:442] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:58:21.107626 10711 tablet_replica.cc:439] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d42d86efb50b4863819800dc62467b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } }
I20260808 03:58:21.107937 10711 tablet_replica.cc:442] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: This TxnStatusTablet replica's current role is: LEADER
I20260808 03:58:21.108371 10656 mvcc.cc:205] Tried to move back new op lower bound from 7316117508395147264 to 7316117508243566592. Current Snapshot: MvccSnapshot[applied={T|T < 7316117508395147264 or (T in {7316117508395147264})}]
I20260808 03:58:21.114060 10639 tablet_replica.cc:439] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d42d86efb50b4863819800dc62467b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } }
I20260808 03:58:21.114456 10639 tablet_replica.cc:442] T 1ade0ff896a74735b3cc33d4ebfc03f7 P bc3134c83cc74c2bb6e2aa70e96af964: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:58:21.129009 10663 tablet_replica.cc:439] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4: TxnStatusTablet state changed. Reason: New leader d42d86efb50b4863819800dc62467b6b. Latest consensus state: current_term: 1 leader_uuid: "d42d86efb50b4863819800dc62467b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } }
I20260808 03:58:21.130497 10663 tablet_replica.cc:442] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:58:21.136274 10663 tablet_replica.cc:439] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d42d86efb50b4863819800dc62467b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } }
I20260808 03:58:21.136550 10663 tablet_replica.cc:442] T 1ade0ff896a74735b3cc33d4ebfc03f7 P 50de18ad8be445029bbadc1679b631b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 03:58:21.138474 10722 tablet_replica.cc:439] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d42d86efb50b4863819800dc62467b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } } }
I20260808 03:58:21.138886 10722 tablet_replica.cc:442] T 1ade0ff896a74735b3cc33d4ebfc03f7 P d42d86efb50b4863819800dc62467b6b: This TxnStatusTablet replica's current role is: LEADER
I20260808 03:58:21.147146 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.125647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.28.251.195:45145)
0808 03:58:21.131537 (+ 5890us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:21.131544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:21.139607 (+ 8063us) server_negotiation.cc:407] Connection header received
0808 03:58:21.139985 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:21.139991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:21.140105 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:21.140247 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:58:21.140966 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.142079 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.144109 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.144823 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.144926 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.145604 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:58:21.146389 (+ 785us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:58:21.146503 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:21.146729 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 03:58:21.146826 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5700}
I20260808 03:58:21.165948 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.159783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:55169 (local address 127.28.251.193:44275)
0808 03:58:21.159977 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:21.159981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:21.160644 (+ 663us) server_negotiation.cc:407] Connection header received
0808 03:58:21.160955 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:21.160959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:21.161036 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:21.161120 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:58:21.161660 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.162464 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.164030 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.164604 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.164717 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.165171 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:58:21.165286 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:58:21.165362 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:21.165509 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 03:58:21.165594 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 03:58:21.165987 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.159697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44275 (local address 127.28.251.195:55169)
0808 03:58:21.160517 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:21.160539 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:21.160807 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:21.161154 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:21.161159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:21.161173 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 03:58:21.161489 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:21.161499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.162603 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.162609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:21.163883 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:21.163899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.164793 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.164799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:21.164928 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.164990 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 03:58:21.165387 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 03:58:21.165398 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 03:58:21.165469 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 03:58:21.165611 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":486,"threads_started":1}
I20260808 03:58:21.179010 10271 raft_consensus.cc:1261] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Refusing update from remote peer bc3134c83cc74c2bb6e2aa70e96af964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:21.179024 10038 raft_consensus.cc:1261] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Refusing update from remote peer bc3134c83cc74c2bb6e2aa70e96af964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:21.179801 10684 consensus_queue.cc:1035] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:21.180179 10639 consensus_queue.cc:1035] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:21.193102 10658 mvcc.cc:205] Tried to move back new op lower bound from 7316117508822056960 to 7316117507590660096. Current Snapshot: MvccSnapshot[applied={T|T < 7316117508822056960}]
I20260808 03:58:21.235750 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.226367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:41453 (local address 127.28.251.195:45145)
0808 03:58:21.226559 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:21.226565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:21.226846 (+ 281us) server_negotiation.cc:407] Connection header received
0808 03:58:21.227163 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:21.227169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:21.227277 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:21.227344 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:58:21.228374 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.229588 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.232963 (+ 3375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.233624 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.233778 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.234402 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:58:21.234562 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:58:21.234625 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:21.235297 (+ 672us) server_negotiation.cc:299] Negotiation successful
0808 03:58:21.235403 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 03:58:21.237097 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.225948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:45145 (local address 127.28.251.193:41453)
0808 03:58:21.226722 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:21.226743 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:21.226988 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:21.227660 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:21.227664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:21.227681 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 03:58:21.228120 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:21.228133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.229755 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.229761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:21.232782 (+ 3021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:21.232800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.233851 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.233859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:21.234050 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.234136 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 03:58:21.235145 (+ 1009us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 03:58:21.235156 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 03:58:21.236448 (+ 1292us) client_negotiation.cc:240] Negotiation successful
0808 03:58:21.236623 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":1631,"threads_started":1}
I20260808 03:58:21.284283 9788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:21.284605 9788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:21.288746 9788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7: Sending DeleteTablet for 3 replicas of tablet 0512b137a859437ba29bc3f706d783af
I20260808 03:58:21.289935 10002 tablet_service.cc:1514] Processing DeleteTablet for tablet 0512b137a859437ba29bc3f706d783af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:21 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:51298
I20260808 03:58:21.290318 10235 tablet_service.cc:1514] Processing DeleteTablet for tablet 0512b137a859437ba29bc3f706d783af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:21 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:33306
I20260808 03:58:21.290812 10486 tablet_service.cc:1514] Processing DeleteTablet for tablet 0512b137a859437ba29bc3f706d783af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:21 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:43422
I20260808 03:58:21.303918 10738 tablet_replica.cc:330] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b: stopping tablet replica
I20260808 03:58:21.304347 10738 raft_consensus.cc:2229] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:21.304760 10738 raft_consensus.cc:2258] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:21.306210 10738 ts_tablet_manager.cc:1905] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:21.314694 10738 ts_tablet_manager.cc:1918] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:58:21.315003 10738 log.cc:1196] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/0512b137a859437ba29bc3f706d783af
I20260808 03:58:21.315755 10738 ts_tablet_manager.cc:1939] T 0512b137a859437ba29bc3f706d783af P d42d86efb50b4863819800dc62467b6b: Deleting consensus metadata
I20260808 03:58:21.317359 9763 catalog_manager.cc:4841] TS d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275): tablet 0512b137a859437ba29bc3f706d783af (table test-table [id=5079146abc3c4f4eb4c9fec322fec24a]) successfully deleted
I20260808 03:58:21.323926 10740 tablet_replica.cc:330] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964: stopping tablet replica
I20260808 03:58:21.324296 10740 raft_consensus.cc:2229] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:58:21.325074 10740 raft_consensus.cc:2258] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:21.325611 10739 tablet_replica.cc:330] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4: stopping tablet replica
I20260808 03:58:21.325886 10739 raft_consensus.cc:2229] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:21.326196 10739 raft_consensus.cc:2258] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:21.328459 10739 ts_tablet_manager.cc:1905] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:21.328491 10740 ts_tablet_manager.cc:1905] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:21.336136 10739 ts_tablet_manager.cc:1918] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:58:21.336401 10739 log.cc:1196] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/0512b137a859437ba29bc3f706d783af
I20260808 03:58:21.336925 10739 ts_tablet_manager.cc:1939] T 0512b137a859437ba29bc3f706d783af P 50de18ad8be445029bbadc1679b631b4: Deleting consensus metadata
I20260808 03:58:21.338251 9765 catalog_manager.cc:4841] TS 50de18ad8be445029bbadc1679b631b4 (127.28.251.194:32979): tablet 0512b137a859437ba29bc3f706d783af (table test-table [id=5079146abc3c4f4eb4c9fec322fec24a]) successfully deleted
I20260808 03:58:21.339751 10740 ts_tablet_manager.cc:1918] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:58:21.340154 10740 log.cc:1196] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/0512b137a859437ba29bc3f706d783af
I20260808 03:58:21.340728 10740 ts_tablet_manager.cc:1939] T 0512b137a859437ba29bc3f706d783af P bc3134c83cc74c2bb6e2aa70e96af964: Deleting consensus metadata
I20260808 03:58:21.342119 9765 catalog_manager.cc:4841] TS bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195:45145): tablet 0512b137a859437ba29bc3f706d783af (table test-table [id=5079146abc3c4f4eb4c9fec322fec24a]) successfully deleted
Aug 08 03:58:21 dist-test-slave-pww0 krb5kdc[9714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 03:58:21 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:21 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 03:58:21.411031 10699 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:21.397686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.28.251.254:43799)
0808 03:58:21.397894 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:21.397900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:21.398022 (+ 122us) server_negotiation.cc:407] Connection header received
0808 03:58:21.398330 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:21.398336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:21.398430 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:21.398547 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:21.400345 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.401524 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.403353 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.404037 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.404188 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.410743 (+ 6555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39070: BlockingRecv error: recv got EOF from 127.0.0.1:39070 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 08 03:58:21 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM
I20260808 03:58:21.437665 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.417748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.28.251.254:43799)
0808 03:58:21.417965 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:21.417971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:21.418530 (+ 559us) server_negotiation.cc:407] Connection header received
0808 03:58:21.418725 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:21.418731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:21.418821 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:21.418933 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:21.420731 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.421929 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.423914 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.424603 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.424744 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.430080 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:21.430088 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:21.430094 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:21.430148 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:21.434000 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:21.435031 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:21.435038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:21.435042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:21.435235 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:21.435894 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:21.435900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:21.435904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:21.436309 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:21.436604 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:21.436999 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 03:58:21.437278 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 03:58:21.442199 9788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:58:21.443049 9788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:58:21.452227 10235 tablet_service.cc:1467] Processing CreateTablet for tablet 138c8cf7dd5f4f97944782453ae99259 (DEFAULT_TABLE table=test-table [id=e4f86c7be25a4654b3aff92f413031c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:21.452414 10486 tablet_service.cc:1467] Processing CreateTablet for tablet 138c8cf7dd5f4f97944782453ae99259 (DEFAULT_TABLE table=test-table [id=e4f86c7be25a4654b3aff92f413031c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:21.452811 10235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138c8cf7dd5f4f97944782453ae99259. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:21.452888 10486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138c8cf7dd5f4f97944782453ae99259. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:21.456275 10002 tablet_service.cc:1467] Processing CreateTablet for tablet 138c8cf7dd5f4f97944782453ae99259 (DEFAULT_TABLE table=test-table [id=e4f86c7be25a4654b3aff92f413031c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:21.456746 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138c8cf7dd5f4f97944782453ae99259. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:21.461498 10634 tablet_bootstrap.cc:492] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4: Bootstrap starting.
I20260808 03:58:21.463404 10634 tablet_bootstrap.cc:654] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:21.469990 10627 tablet_bootstrap.cc:492] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b: Bootstrap starting.
I20260808 03:58:21.472034 10627 tablet_bootstrap.cc:654] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:21.476065 10626 tablet_bootstrap.cc:492] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964: Bootstrap starting.
I20260808 03:58:21.476830 10634 tablet_bootstrap.cc:492] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4: No bootstrap required, opened a new log
I20260808 03:58:21.477012 10634 ts_tablet_manager.cc:1397] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.002s
I20260808 03:58:21.477939 10634 raft_consensus.cc:348] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.478463 10626 tablet_bootstrap.cc:654] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:21.479674 10634 raft_consensus.cc:374] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:21.479825 10634 raft_consensus.cc:729] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50de18ad8be445029bbadc1679b631b4, State: Initialized, Role: FOLLOWER
I20260808 03:58:21.480137 10634 consensus_queue.cc:260] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.480912 10634 ts_tablet_manager.cc:1428] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260808 03:58:21.485344 10626 tablet_bootstrap.cc:492] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964: No bootstrap required, opened a new log
I20260808 03:58:21.485581 10626 ts_tablet_manager.cc:1397] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260808 03:58:21.486477 10626 raft_consensus.cc:348] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.486791 10626 raft_consensus.cc:374] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:21.486923 10626 raft_consensus.cc:729] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3134c83cc74c2bb6e2aa70e96af964, State: Initialized, Role: FOLLOWER
I20260808 03:58:21.487181 10626 consensus_queue.cc:260] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.487962 10626 ts_tablet_manager.cc:1428] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 03:58:21.490861 10627 tablet_bootstrap.cc:492] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b: No bootstrap required, opened a new log
I20260808 03:58:21.491035 10627 ts_tablet_manager.cc:1397] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.000s
I20260808 03:58:21.491896 10627 raft_consensus.cc:348] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.492162 10627 raft_consensus.cc:374] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:21.492327 10627 raft_consensus.cc:729] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42d86efb50b4863819800dc62467b6b, State: Initialized, Role: FOLLOWER
I20260808 03:58:21.492630 10627 consensus_queue.cc:260] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.494275 10627 ts_tablet_manager.cc:1428] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:21.611749 10663 raft_consensus.cc:482] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:58:21.611994 10663 raft_consensus.cc:504] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.612744 10663 leader_election.cc:288] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275), bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195:45145)
I20260808 03:58:21.613507 10515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138c8cf7dd5f4f97944782453ae99259" candidate_uuid: "50de18ad8be445029bbadc1679b631b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" is_pre_election: true
I20260808 03:58:21.613523 10038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138c8cf7dd5f4f97944782453ae99259" candidate_uuid: "50de18ad8be445029bbadc1679b631b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42d86efb50b4863819800dc62467b6b" is_pre_election: true
I20260808 03:58:21.613793 10515 raft_consensus.cc:2454] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50de18ad8be445029bbadc1679b631b4 in term 0.
I20260808 03:58:21.613827 10038 raft_consensus.cc:2454] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50de18ad8be445029bbadc1679b631b4 in term 0.
I20260808 03:58:21.614351 10156 leader_election.cc:302] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50de18ad8be445029bbadc1679b631b4, bc3134c83cc74c2bb6e2aa70e96af964; no voters:
I20260808 03:58:21.614697 10663 raft_consensus.cc:2781] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:58:21.614821 10663 raft_consensus.cc:482] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:58:21.614914 10663 raft_consensus.cc:3037] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:21.617494 10663 raft_consensus.cc:504] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.618149 10663 leader_election.cc:288] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [CANDIDATE]: Term 1 election: Requested vote from peers d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275), bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195:45145)
I20260808 03:58:21.618775 10515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138c8cf7dd5f4f97944782453ae99259" candidate_uuid: "50de18ad8be445029bbadc1679b631b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3134c83cc74c2bb6e2aa70e96af964"
I20260808 03:58:21.619017 10515 raft_consensus.cc:3037] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:21.619093 10038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138c8cf7dd5f4f97944782453ae99259" candidate_uuid: "50de18ad8be445029bbadc1679b631b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42d86efb50b4863819800dc62467b6b"
I20260808 03:58:21.619390 10038 raft_consensus.cc:3037] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:21.622615 10038 raft_consensus.cc:2454] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50de18ad8be445029bbadc1679b631b4 in term 1.
I20260808 03:58:21.622632 10515 raft_consensus.cc:2454] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50de18ad8be445029bbadc1679b631b4 in term 1.
I20260808 03:58:21.623216 10156 leader_election.cc:302] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50de18ad8be445029bbadc1679b631b4, bc3134c83cc74c2bb6e2aa70e96af964; no voters:
I20260808 03:58:21.623654 10663 raft_consensus.cc:2781] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:21.624130 10663 raft_consensus.cc:686] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 1 LEADER]: Becoming Leader. State: Replica: 50de18ad8be445029bbadc1679b631b4, State: Running, Role: LEADER
I20260808 03:58:21.624473 10663 consensus_queue.cc:237] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.628007 9788 catalog_manager.cc:5495] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50de18ad8be445029bbadc1679b631b4 (127.28.251.194). New cstate: current_term: 1 leader_uuid: "50de18ad8be445029bbadc1679b631b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } health_report { overall_health: UNKNOWN } } }
I20260808 03:58:21.671375 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.664618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.28.251.194:32979)
0808 03:58:21.665199 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:21.665205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:21.665227 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:58:21.665303 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:21.665308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:21.665403 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:21.665507 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:58:21.665934 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.667029 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.668969 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.669662 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.669789 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.670261 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:58:21.670451 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:58:21.670557 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:21.670703 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 03:58:21.670796 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":283,"threads_started":1}
I20260808 03:58:21.674643 10038 raft_consensus.cc:1261] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Refusing update from remote peer 50de18ad8be445029bbadc1679b631b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:21.674688 10515 raft_consensus.cc:1261] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Refusing update from remote peer 50de18ad8be445029bbadc1679b631b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:21.675514 10761 consensus_queue.cc:1035] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:21.676159 10663 consensus_queue.cc:1035] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:21.695884 9785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:21.696151 9785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:21.699769 9785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7: Sending DeleteTablet for 3 replicas of tablet 138c8cf7dd5f4f97944782453ae99259
I20260808 03:58:21.700660 10235 tablet_service.cc:1514] Processing DeleteTablet for tablet 138c8cf7dd5f4f97944782453ae99259 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:21 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:33306
I20260808 03:58:21.700672 10002 tablet_service.cc:1514] Processing DeleteTablet for tablet 138c8cf7dd5f4f97944782453ae99259 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:21 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:51298
I20260808 03:58:21.700913 10486 tablet_service.cc:1514] Processing DeleteTablet for tablet 138c8cf7dd5f4f97944782453ae99259 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:21 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:43422
I20260808 03:58:21.701033 10738 tablet_replica.cc:330] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b: stopping tablet replica
I20260808 03:58:21.701270 10738 raft_consensus.cc:2229] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:21.701543 10738 raft_consensus.cc:2258] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:21.701870 10739 tablet_replica.cc:330] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4: stopping tablet replica
I20260808 03:58:21.702090 10739 raft_consensus.cc:2229] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:58:21.702450 10739 raft_consensus.cc:2258] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:21.702783 10740 tablet_replica.cc:330] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964: stopping tablet replica
I20260808 03:58:21.702864 10738 ts_tablet_manager.cc:1905] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:21.703016 10740 raft_consensus.cc:2229] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:21.703281 10740 raft_consensus.cc:2258] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:21.703218 9785 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:58:21.704480 9785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:58:21.708113 10739 ts_tablet_manager.cc:1905] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:21.715373 10739 ts_tablet_manager.cc:1918] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:21.715579 10739 log.cc:1196] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/138c8cf7dd5f4f97944782453ae99259
I20260808 03:58:21.717072 10235 tablet_service.cc:1467] Processing CreateTablet for tablet 8eef9facd73c4525a1c80c0fdb5fb5b2 (DEFAULT_TABLE table=test-table [id=d4ea72753117462db73859bb37a8e7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:21.717113 10486 tablet_service.cc:1467] Processing CreateTablet for tablet 8eef9facd73c4525a1c80c0fdb5fb5b2 (DEFAULT_TABLE table=test-table [id=d4ea72753117462db73859bb37a8e7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:21.717582 10235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eef9facd73c4525a1c80c0fdb5fb5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:21.717675 10486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eef9facd73c4525a1c80c0fdb5fb5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:21.718995 10738 ts_tablet_manager.cc:1918] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:21.719151 10738 log.cc:1196] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/138c8cf7dd5f4f97944782453ae99259
I20260808 03:58:21.719638 10738 ts_tablet_manager.cc:1939] T 138c8cf7dd5f4f97944782453ae99259 P d42d86efb50b4863819800dc62467b6b: Deleting consensus metadata
I20260808 03:58:21.721238 10739 ts_tablet_manager.cc:1939] T 138c8cf7dd5f4f97944782453ae99259 P 50de18ad8be445029bbadc1679b631b4: Deleting consensus metadata
I20260808 03:58:21.722589 9765 catalog_manager.cc:4841] TS 50de18ad8be445029bbadc1679b631b4 (127.28.251.194:32979): tablet 138c8cf7dd5f4f97944782453ae99259 (table test-table [id=e4f86c7be25a4654b3aff92f413031c0]) successfully deleted
I20260808 03:58:21.723734 10740 ts_tablet_manager.cc:1905] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:21.723861 9763 catalog_manager.cc:4841] TS d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275): tablet 138c8cf7dd5f4f97944782453ae99259 (table test-table [id=e4f86c7be25a4654b3aff92f413031c0]) successfully deleted
I20260808 03:58:21.724777 10002 tablet_service.cc:1467] Processing CreateTablet for tablet 8eef9facd73c4525a1c80c0fdb5fb5b2 (DEFAULT_TABLE table=test-table [id=d4ea72753117462db73859bb37a8e7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:21.725311 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eef9facd73c4525a1c80c0fdb5fb5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:21.726548 10626 tablet_bootstrap.cc:492] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964: Bootstrap starting.
I20260808 03:58:21.728557 10626 tablet_bootstrap.cc:654] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:21.731182 10740 ts_tablet_manager.cc:1918] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:21.731380 10740 log.cc:1196] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/138c8cf7dd5f4f97944782453ae99259
I20260808 03:58:21.733155 10740 ts_tablet_manager.cc:1939] T 138c8cf7dd5f4f97944782453ae99259 P bc3134c83cc74c2bb6e2aa70e96af964: Deleting consensus metadata
I20260808 03:58:21.733487 10634 tablet_bootstrap.cc:492] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4: Bootstrap starting.
I20260808 03:58:21.734455 9765 catalog_manager.cc:4841] TS bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195:45145): tablet 138c8cf7dd5f4f97944782453ae99259 (table test-table [id=e4f86c7be25a4654b3aff92f413031c0]) successfully deleted
I20260808 03:58:21.735491 10634 tablet_bootstrap.cc:654] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:21.737414 10627 tablet_bootstrap.cc:492] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b: Bootstrap starting.
I20260808 03:58:21.738417 10626 tablet_bootstrap.cc:492] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964: No bootstrap required, opened a new log
I20260808 03:58:21.738682 10626 ts_tablet_manager.cc:1397] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260808 03:58:21.739220 10627 tablet_bootstrap.cc:654] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:21.739806 10626 raft_consensus.cc:348] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.740073 10626 raft_consensus.cc:374] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:21.740166 10626 raft_consensus.cc:729] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3134c83cc74c2bb6e2aa70e96af964, State: Initialized, Role: FOLLOWER
I20260808 03:58:21.740465 10626 consensus_queue.cc:260] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.741039 10626 ts_tablet_manager.cc:1428] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 03:58:21.741322 10634 tablet_bootstrap.cc:492] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4: No bootstrap required, opened a new log
I20260808 03:58:21.741513 10634 ts_tablet_manager.cc:1397] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260808 03:58:21.742417 10634 raft_consensus.cc:348] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.742683 10634 raft_consensus.cc:374] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:21.742763 10634 raft_consensus.cc:729] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50de18ad8be445029bbadc1679b631b4, State: Initialized, Role: FOLLOWER
I20260808 03:58:21.743083 10634 consensus_queue.cc:260] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.743924 10634 ts_tablet_manager.cc:1428] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 03:58:21.745676 10627 tablet_bootstrap.cc:492] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b: No bootstrap required, opened a new log
I20260808 03:58:21.745885 10627 ts_tablet_manager.cc:1397] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20260808 03:58:21.746721 10627 raft_consensus.cc:348] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.747014 10627 raft_consensus.cc:374] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:21.747112 10627 raft_consensus.cc:729] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42d86efb50b4863819800dc62467b6b, State: Initialized, Role: FOLLOWER
I20260808 03:58:21.747401 10627 consensus_queue.cc:260] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.748988 10627 ts_tablet_manager.cc:1428] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 03:58:21.753741 10684 raft_consensus.cc:482] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:58:21.753968 10684 raft_consensus.cc:504] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.754639 10684 leader_election.cc:288] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275), 50de18ad8be445029bbadc1679b631b4 (127.28.251.194:32979)
I20260808 03:58:21.755136 10038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eef9facd73c4525a1c80c0fdb5fb5b2" candidate_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42d86efb50b4863819800dc62467b6b" is_pre_election: true
I20260808 03:58:21.755360 10038 raft_consensus.cc:2454] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc3134c83cc74c2bb6e2aa70e96af964 in term 0.
I20260808 03:58:21.755280 10272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eef9facd73c4525a1c80c0fdb5fb5b2" candidate_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50de18ad8be445029bbadc1679b631b4" is_pre_election: true
I20260808 03:58:21.755625 10272 raft_consensus.cc:2454] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc3134c83cc74c2bb6e2aa70e96af964 in term 0.
I20260808 03:58:21.755880 10394 leader_election.cc:302] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc3134c83cc74c2bb6e2aa70e96af964, d42d86efb50b4863819800dc62467b6b; no voters:
I20260808 03:58:21.756209 10684 raft_consensus.cc:2781] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:58:21.756301 10684 raft_consensus.cc:482] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:58:21.756422 10684 raft_consensus.cc:3037] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:21.758889 10684 raft_consensus.cc:504] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.760004 10684 leader_election.cc:288] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [CANDIDATE]: Term 1 election: Requested vote from peers d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275), 50de18ad8be445029bbadc1679b631b4 (127.28.251.194:32979)
I20260808 03:58:21.760110 10038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eef9facd73c4525a1c80c0fdb5fb5b2" candidate_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42d86efb50b4863819800dc62467b6b"
I20260808 03:58:21.760303 10038 raft_consensus.cc:3037] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:21.760322 10272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eef9facd73c4525a1c80c0fdb5fb5b2" candidate_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50de18ad8be445029bbadc1679b631b4"
I20260808 03:58:21.760578 10272 raft_consensus.cc:3037] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:21.763208 10038 raft_consensus.cc:2454] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc3134c83cc74c2bb6e2aa70e96af964 in term 1.
I20260808 03:58:21.763720 10394 leader_election.cc:302] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc3134c83cc74c2bb6e2aa70e96af964, d42d86efb50b4863819800dc62467b6b; no voters:
I20260808 03:58:21.763736 10272 raft_consensus.cc:2454] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc3134c83cc74c2bb6e2aa70e96af964 in term 1.
I20260808 03:58:21.764052 10684 raft_consensus.cc:2781] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:21.764570 10684 raft_consensus.cc:686] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 LEADER]: Becoming Leader. State: Replica: bc3134c83cc74c2bb6e2aa70e96af964, State: Running, Role: LEADER
I20260808 03:58:21.764896 10684 consensus_queue.cc:237] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } }
I20260808 03:58:21.768082 9788 catalog_manager.cc:5495] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195). New cstate: current_term: 1 leader_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc3134c83cc74c2bb6e2aa70e96af964" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45145 } health_report { overall_health: HEALTHY } } }
I20260808 03:58:21.800308 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.793471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.28.251.195:45145)
0808 03:58:21.794113 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:21.794119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:21.794139 (+ 20us) server_negotiation.cc:407] Connection header received
0808 03:58:21.794212 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:21.794217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:21.794286 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:21.794383 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:58:21.794927 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.795991 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.798121 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.798642 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.798756 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.799366 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:58:21.799551 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:58:21.799625 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:21.799822 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 03:58:21.799910 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":593,"threads_started":1}
I20260808 03:58:21.812419 10038 raft_consensus.cc:1261] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Refusing update from remote peer bc3134c83cc74c2bb6e2aa70e96af964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:21.812438 10272 raft_consensus.cc:1261] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Refusing update from remote peer bc3134c83cc74c2bb6e2aa70e96af964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:21.813108 10775 consensus_queue.cc:1035] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50de18ad8be445029bbadc1679b631b4" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:21.813565 10684 consensus_queue.cc:1035] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42d86efb50b4863819800dc62467b6b" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:21.864957 9783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:21.865187 9783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:21.868155 9783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7623b54c346448dfacb85b3389d119f7: Sending DeleteTablet for 3 replicas of tablet 8eef9facd73c4525a1c80c0fdb5fb5b2
I20260808 03:58:21.869071 10235 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eef9facd73c4525a1c80c0fdb5fb5b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:21 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:33306
I20260808 03:58:21.869204 10486 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eef9facd73c4525a1c80c0fdb5fb5b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:21 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:43422
I20260808 03:58:21.869447 10739 tablet_replica.cc:330] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4: stopping tablet replica
I20260808 03:58:21.869509 10740 tablet_replica.cc:330] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964: stopping tablet replica
I20260808 03:58:21.869668 10740 raft_consensus.cc:2229] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:58:21.869661 10739 raft_consensus.cc:2229] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:21.869897 10739 raft_consensus.cc:2258] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:21.870028 10740 raft_consensus.cc:2258] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:21.870066 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 9902
I20260808 03:58:21.870533 10002 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eef9facd73c4525a1c80c0fdb5fb5b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:21 UTC) from {username='oryx', principal='oryx/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:51298
I20260808 03:58:21.871428 10739 ts_tablet_manager.cc:1905] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:21.871860 10740 ts_tablet_manager.cc:1905] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:21.873027 10738 tablet_replica.cc:330] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b: stopping tablet replica
I20260808 03:58:21.873240 10738 raft_consensus.cc:2229] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:21.873461 10738 raft_consensus.cc:2258] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:21.874946 10738 ts_tablet_manager.cc:1905] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:21.878847 10740 ts_tablet_manager.cc:1918] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:58:21.878860 10739 ts_tablet_manager.cc:1918] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:58:21.878999 10739 log.cc:1196] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/8eef9facd73c4525a1c80c0fdb5fb5b2
I20260808 03:58:21.879007 10740 log.cc:1196] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/8eef9facd73c4525a1c80c0fdb5fb5b2
I20260808 03:58:21.879426 10740 ts_tablet_manager.cc:1939] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P bc3134c83cc74c2bb6e2aa70e96af964: Deleting consensus metadata
I20260808 03:58:21.879436 10739 ts_tablet_manager.cc:1939] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P 50de18ad8be445029bbadc1679b631b4: Deleting consensus metadata
I20260808 03:58:21.880699 9765 catalog_manager.cc:4841] TS 50de18ad8be445029bbadc1679b631b4 (127.28.251.194:32979): tablet 8eef9facd73c4525a1c80c0fdb5fb5b2 (table test-table [id=d4ea72753117462db73859bb37a8e7b4]) successfully deleted
I20260808 03:58:21.881052 9765 catalog_manager.cc:4841] TS bc3134c83cc74c2bb6e2aa70e96af964 (127.28.251.195:45145): tablet 8eef9facd73c4525a1c80c0fdb5fb5b2 (table test-table [id=d4ea72753117462db73859bb37a8e7b4]) successfully deleted
I20260808 03:58:21.882195 10738 ts_tablet_manager.cc:1918] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 03:58:21.882329 10738 log.cc:1196] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/8eef9facd73c4525a1c80c0fdb5fb5b2
I20260808 03:58:21.882725 10738 ts_tablet_manager.cc:1939] T 8eef9facd73c4525a1c80c0fdb5fb5b2 P d42d86efb50b4863819800dc62467b6b: Deleting consensus metadata
I20260808 03:58:21.883710 9763 catalog_manager.cc:4841] TS d42d86efb50b4863819800dc62467b6b (127.28.251.193:44275): tablet 8eef9facd73c4525a1c80c0fdb5fb5b2 (table test-table [id=d4ea72753117462db73859bb37a8e7b4]) successfully deleted
Aug 08 03:58:21 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.251.193@KRBTEST.COM
I20260808 03:58:21.890548 10794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:21.871569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.28.251.193:44275)
0808 03:58:21.872805 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:21.872811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:21.874137 (+ 1326us) server_negotiation.cc:407] Connection header received
0808 03:58:21.874422 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:21.874428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:21.874510 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:21.874584 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:21.876474 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.877745 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.879575 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:21.880250 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:21.880384 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:21.885093 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:21.885111 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:21.885128 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:21.885165 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:21.887839 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:21.888582 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:21.888589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:21.888592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:21.888723 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:21.889119 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:21.889122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:21.889125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:21.889511 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:21.889772 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:21.890060 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 03:58:21.890250 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":960,"threads_started":1}
I20260808 03:58:21.891323 10114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:22.020448 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 9902
I20260808 03:58:22.061015 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 10126
Aug 08 03:58:22 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.251.194@KRBTEST.COM
I20260808 03:58:22.077724 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:22.061853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.28.251.194:32979)
0808 03:58:22.062082 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:22.062087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:22.062107 (+ 20us) server_negotiation.cc:407] Connection header received
0808 03:58:22.062281 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:22.062286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:22.062359 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:22.062455 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:22.063970 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:22.064777 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:22.066174 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:22.066598 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:22.066728 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:22.071303 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:22.071324 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:22.071345 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:22.071394 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:22.074563 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:22.075372 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:22.075381 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:22.075383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:22.075693 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:22.076108 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:22.076112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:22.076114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:22.076511 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:22.076855 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:22.077158 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 03:58:22.077400 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260808 03:58:22.078397 10345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:22.226020 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 10126
I20260808 03:58:22.260545 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 10366
Aug 08 03:58:22 dist-test-slave-pww0 krb5kdc[9714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.251.195@KRBTEST.COM
I20260808 03:58:22.276469 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:22.261381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42542 (local address 127.28.251.195:45145)
0808 03:58:22.261624 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:22.261631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:22.261748 (+ 117us) server_negotiation.cc:407] Connection header received
0808 03:58:22.261962 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:22.261966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:22.262038 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:22.262148 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:22.263521 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:22.264230 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:22.265478 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:22.265940 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:22.266055 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:22.270366 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:22.270386 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:22.270405 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:22.270446 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:22.273392 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:22.274131 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:22.274139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:22.274142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:22.274300 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:22.274681 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:22.274686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:22.274689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:22.275067 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:22.275401 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:22.275716 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 03:58:22.275983 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260808 03:58:22.277170 10581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:22.407953 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 10366
I20260808 03:58:22.443387 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 9737
I20260808 03:58:22.444718 9849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:22.566030 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 9737
2026-08-08T03:58:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6387 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:58:22 dist-test-slave-pww0 krb5kdc[10808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:58:22 dist-test-slave-pww0 krb5kdc[10808](info): set up 2 sockets
Aug 08 03:58:22 dist-test-slave-pww0 krb5kdc[10808](info): commencing operation
krb5kdc: starting...
W20260808 03:58:23.843993 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.007s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:58:24 dist-test-slave-pww0 krb5kdc[10808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:58:24.158898 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:46019
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:33267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:46019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:24.525705 10833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:24.526050 10833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:24.526196 10833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:24.533748 10833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:58:24.533886 10833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:24.533967 10833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:24.534050 10833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:58:24.534112 10833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:58:24.544857 10833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:33267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:46019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:46019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:24.546343 10833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:24.547828 10833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:24.560570 10843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:24.560654 10844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:24.560693 10848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:24.561643 10833 server_base.cc:1034] running on GCE node
I20260808 03:58:24.562476 10833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:24.564081 10833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:24.565310 10833 hybrid_clock.cc:648] HybridClock initialized: now 1786161504565251 us; error 51 us; skew 500 ppm
I20260808 03:58:24.565697 10833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T03:58:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2069 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:58:24 dist-test-slave-pww0 krb5kdc[10860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:58:24 dist-test-slave-pww0 krb5kdc[10860](info): set up 2 sockets
Aug 08 03:58:24 dist-test-slave-pww0 krb5kdc[10860](info): commencing operation
krb5kdc: starting...
W20260808 03:58:25.912264 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.003s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:58:26 dist-test-slave-pww0 krb5kdc[10860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:58:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:58:26.219220 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:37713
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:37713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:26.578955 10883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:26.579276 10883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:26.579368 10883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:26.587263 10883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:58:26.587393 10883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:26.587494 10883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:26.587582 10883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:58:26.587648 10883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:58:26.598096 10883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:37713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:37713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:26.599597 10883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:26.601022 10883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:26.613577 10890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:26.613602 10891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:26.613669 10895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:26.617422 10883 server_base.cc:1034] running on GCE node
I20260808 03:58:26.618345 10883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:26.619928 10883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:26.621151 10883 hybrid_clock.cc:648] HybridClock initialized: now 1786161506621082 us; error 51 us; skew 500 ppm
I20260808 03:58:26.621516 10883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:26 dist-test-slave-pww0 krb5kdc[10860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161506, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:26.624739 10883 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:58:26.626361 10883 webserver.cc:460] Webserver started at http://127.28.251.254:39325/ using document root <none> and password file <none>
I20260808 03:58:26.627113 10883 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:26.627221 10883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:26.627570 10883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:26.630867 10883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "f5ead044f95f4a3c842fbd654241e37d"
format_stamp: "Formatted at 2026-08-08 03:58:26 on dist-test-slave-pww0"
server_key: "3c355e0df355c3d3bcae0a8ea8837f7d"
server_key_iv: "99b0089afbe04b8936b3094e41f2e57c"
server_key_version: "encryptionkey@0"
I20260808 03:58:26.631712 10883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "f5ead044f95f4a3c842fbd654241e37d"
format_stamp: "Formatted at 2026-08-08 03:58:26 on dist-test-slave-pww0"
server_key: "3c355e0df355c3d3bcae0a8ea8837f7d"
server_key_iv: "99b0089afbe04b8936b3094e41f2e57c"
server_key_version: "encryptionkey@0"
I20260808 03:58:26.637279 10883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 03:58:26.641186 10909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:26.642685 10883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 03:58:26.642866 10883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "f5ead044f95f4a3c842fbd654241e37d"
format_stamp: "Formatted at 2026-08-08 03:58:26 on dist-test-slave-pww0"
server_key: "3c355e0df355c3d3bcae0a8ea8837f7d"
server_key_iv: "99b0089afbe04b8936b3094e41f2e57c"
server_key_version: "encryptionkey@0"
I20260808 03:58:26.643074 10883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:26.672636 10883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:26.677341 10883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:26.677645 10883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:26.696841 10883 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:37713
I20260808 03:58:26.696872 10979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:37713 every 8 connection(s)
I20260808 03:58:26.698390 10883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:58:26.701964 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 10883
I20260808 03:58:26.702205 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:58:26.702950 10980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:26.703070 29679 external_mini_cluster.cc:1428] Setting key 161f7427d97fe9f9968420a482a95557
Aug 08 03:58:26 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:26.719436 10980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d: Bootstrap starting.
I20260808 03:58:26.723237 10980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:26.724484 10980 log.cc:824] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:26.728161 10980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d: No bootstrap required, opened a new log
I20260808 03:58:26.728696 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:26.705404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.28.251.254:37713)
0808 03:58:26.706289 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:26.706302 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:26.706361 (+ 59us) server_negotiation.cc:407] Connection header received
0808 03:58:26.708327 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:26.708355 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:26.708830 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:26.709207 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:26.711306 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:26.712763 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:26.714680 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:26.715568 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:26.715735 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:26.720696 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:26.720706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:26.720723 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:26.720775 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:26.724028 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:26.724904 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:26.724912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:26.724919 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:26.725199 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:26.725672 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:26.725677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:26.725681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:26.726365 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:26.726603 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:26.727137 (+ 534us) server_negotiation.cc:299] Negotiation successful
0808 03:58:26.727545 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":380,"threads_started":1}
I20260808 03:58:26.736091 10980 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } }
I20260808 03:58:26.736533 10980 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:26.736641 10980 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ead044f95f4a3c842fbd654241e37d, State: Initialized, Role: FOLLOWER
I20260808 03:58:26.737232 10980 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } }
I20260808 03:58:26.737483 10980 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:58:26.737622 10980 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:58:26.737789 10980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:26.740188 10980 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } }
I20260808 03:58:26.740746 10980 leader_election.cc:302] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5ead044f95f4a3c842fbd654241e37d; no voters:
I20260808 03:58:26.741408 10980 leader_election.cc:288] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:58:26.741518 10990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:26.742281 10990 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 1 LEADER]: Becoming Leader. State: Replica: f5ead044f95f4a3c842fbd654241e37d, State: Running, Role: LEADER
I20260808 03:58:26.742898 10990 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } }
I20260808 03:58:26.743891 10980 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:58:26.745746 10992 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5ead044f95f4a3c842fbd654241e37d. Latest consensus state: current_term: 1 leader_uuid: "f5ead044f95f4a3c842fbd654241e37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } } }
I20260808 03:58:26.745756 10991 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5ead044f95f4a3c842fbd654241e37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } } }
I20260808 03:58:26.746229 10991 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:26.746341 10992 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:26.747159 10998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:58:26.753024 10998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:58:26.764271 10998 catalog_manager.cc:1347] Generated new cluster ID: 68e8fe1d675b428e9c6fb14368a28bd1
I20260808 03:58:26.764405 10998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:58:26.801208 10998 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:58:26.802620 10998 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:58:26.812964 10998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d: Generated new TSK 0
I20260808 03:58:26.814186 10998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:58:26.907078 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37713
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:27.283499 11028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:27.283838 11028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:27.283972 11028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:27.291587 11028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:27.291723 11028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:27.291893 11028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:58:27.302544 11028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:27.304123 11028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:27.305564 11028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:27.319665 11035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:27.319726 11041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:27.319762 11037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:27.320741 11028 server_base.cc:1034] running on GCE node
I20260808 03:58:27.321471 11028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:27.322572 11028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:27.323813 11028 hybrid_clock.cc:648] HybridClock initialized: now 1786161507323763 us; error 38 us; skew 500 ppm
I20260808 03:58:27.324172 11028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:27 dist-test-slave-pww0 krb5kdc[10860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161507, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:27.327750 11028 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:58:27.329707 11028 webserver.cc:460] Webserver started at http://127.28.251.193:46869/ using document root <none> and password file <none>
I20260808 03:58:27.330526 11028 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:27.330662 11028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:27.331004 11028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:27.334332 11028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "03aea75666404ddc8b33d67b65822c75"
format_stamp: "Formatted at 2026-08-08 03:58:27 on dist-test-slave-pww0"
server_key: "8ddbd3086708ddc9a24dd1b5d86f6b6a"
server_key_iv: "2a94c0b5d895ca20cdfad5c940954b66"
server_key_version: "encryptionkey@0"
I20260808 03:58:27.335250 11028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "03aea75666404ddc8b33d67b65822c75"
format_stamp: "Formatted at 2026-08-08 03:58:27 on dist-test-slave-pww0"
server_key: "8ddbd3086708ddc9a24dd1b5d86f6b6a"
server_key_iv: "2a94c0b5d895ca20cdfad5c940954b66"
server_key_version: "encryptionkey@0"
I20260808 03:58:27.341145 11028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 03:58:27.345257 11059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:27.346894 11028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 03:58:27.347082 11028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "03aea75666404ddc8b33d67b65822c75"
format_stamp: "Formatted at 2026-08-08 03:58:27 on dist-test-slave-pww0"
server_key: "8ddbd3086708ddc9a24dd1b5d86f6b6a"
server_key_iv: "2a94c0b5d895ca20cdfad5c940954b66"
server_key_version: "encryptionkey@0"
I20260808 03:58:27.347301 11028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:27.362826 11028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:27.367587 11028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:27.367894 11028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:27.368961 11028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:27.370537 11028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:27.370626 11028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:27.370752 11028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:27.370812 11028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:27.404472 11028 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:39109
I20260808 03:58:27.404522 11263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:39109 every 8 connection(s)
I20260808 03:58:27.405905 11028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:58:27.411536 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 11028
I20260808 03:58:27.411738 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:58:27.412393 29679 external_mini_cluster.cc:1428] Setting key a7f1f9224d22f7e38867fb9ff2454140
Aug 08 03:58:27 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161507, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:27.427961 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:27.408769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:56343 (local address 127.28.251.254:37713)
0808 03:58:27.409261 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:27.409266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:27.410490 (+ 1224us) server_negotiation.cc:407] Connection header received
0808 03:58:27.412613 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:27.412620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:27.412716 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:27.412821 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:27.414803 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:27.415569 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:27.417230 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:27.417657 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:27.417778 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:27.421625 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:27.421631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:27.421634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:27.421675 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:27.424456 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:27.425345 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:27.425351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:27.425353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:27.425504 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:27.426000 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:27.426005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:27.426007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:27.426323 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:27.426483 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:27.427282 (+ 799us) server_negotiation.cc:299] Negotiation successful
0808 03:58:27.427585 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":259,"threads_started":1}
I20260808 03:58:27.429045 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:27.409404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:56343)
0808 03:58:27.410252 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:27.410318 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:27.412282 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:27.413113 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:27.413122 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:27.413850 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:27.414616 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:27.414638 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:27.415741 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:27.415745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:27.417071 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:27.417086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:27.417921 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:27.417929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:27.418172 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:27.418908 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:27.418935 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:27.421343 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:27.424656 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:27.424665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:27.424681 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:27.425196 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:27.425633 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:27.425637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:27.425639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:27.425876 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:27.426469 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:27.426478 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:27.426874 (+ 396us) client_negotiation.cc:769] Sending connection context
0808 03:58:27.427138 (+ 264us) client_negotiation.cc:240] Negotiation successful
0808 03:58:27.427791 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":354,"threads_started":1}
I20260808 03:58:27.430859 11264 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37713
I20260808 03:58:27.431165 11264 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:27.431995 11264 heartbeater.cc:511] Master 127.28.251.254:37713 requested a full tablet report, sending...
I20260808 03:58:27.434571 10932 ts_manager.cc:194] Registered new tserver with Master: 03aea75666404ddc8b33d67b65822c75 (127.28.251.193:39109)
I20260808 03:58:27.436625 10932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:56343
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:58:27.482385 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37713
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:27.844519 11272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:27.844887 11272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:27.845041 11272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:27.852785 11272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:27.852926 11272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:27.853091 11272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:58:27.864101 11272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:27.865605 11272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:27.867028 11272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:27.881361 11287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:27.881402 11280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:27.881592 11279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:27.882364 11272 server_base.cc:1034] running on GCE node
I20260808 03:58:27.883045 11272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:27.884176 11272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:27.885394 11272 hybrid_clock.cc:648] HybridClock initialized: now 1786161507885341 us; error 49 us; skew 500 ppm
I20260808 03:58:27.885741 11272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:27 dist-test-slave-pww0 krb5kdc[10860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161507, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:27.889185 11272 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:58:27.890853 11272 webserver.cc:460] Webserver started at http://127.28.251.194:35987/ using document root <none> and password file <none>
I20260808 03:58:27.891685 11272 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:27.891830 11272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:27.892172 11272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:27.895437 11272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "fcb25bcf21b14c69b35daa3fc2237669"
format_stamp: "Formatted at 2026-08-08 03:58:27 on dist-test-slave-pww0"
server_key: "fb968ac4f1ba4f46a0dae22a3578f44e"
server_key_iv: "3bd92c7c8a6d475aa479ef1f8fe84c7b"
server_key_version: "encryptionkey@0"
I20260808 03:58:27.896375 11272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "fcb25bcf21b14c69b35daa3fc2237669"
format_stamp: "Formatted at 2026-08-08 03:58:27 on dist-test-slave-pww0"
server_key: "fb968ac4f1ba4f46a0dae22a3578f44e"
server_key_iv: "3bd92c7c8a6d475aa479ef1f8fe84c7b"
server_key_version: "encryptionkey@0"
I20260808 03:58:27.902240 11272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 03:58:27.906199 11303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:27.907740 11272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 03:58:27.907932 11272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "fcb25bcf21b14c69b35daa3fc2237669"
format_stamp: "Formatted at 2026-08-08 03:58:27 on dist-test-slave-pww0"
server_key: "fb968ac4f1ba4f46a0dae22a3578f44e"
server_key_iv: "3bd92c7c8a6d475aa479ef1f8fe84c7b"
server_key_version: "encryptionkey@0"
I20260808 03:58:27.908164 11272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:27.920475 11272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:27.924957 11272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:27.925262 11272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:27.926339 11272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:27.927860 11272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:27.927939 11272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:27.928058 11272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:27.928109 11272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:27.961673 11272 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:36251
I20260808 03:58:27.961722 11504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:36251 every 8 connection(s)
I20260808 03:58:27.963096 11272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:58:27.964290 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 11272
I20260808 03:58:27.964480 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:58:27.965083 29679 external_mini_cluster.cc:1428] Setting key d1bca0eedb90656c8af0c8001f52de64
Aug 08 03:58:27 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161507, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:27.988776 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:27.966183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:33577 (local address 127.28.251.254:37713)
0808 03:58:27.966675 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:27.966682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:27.968308 (+ 1626us) server_negotiation.cc:407] Connection header received
0808 03:58:27.970921 (+ 2613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:27.970925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:27.971001 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:27.971099 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:27.973337 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:27.974077 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:27.976420 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:27.976882 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:27.976995 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:27.981599 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:27.981607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:27.981613 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:27.981667 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:27.985207 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:27.986151 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:27.986158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:27.986161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:27.986339 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:27.986800 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:27.986805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:27.986808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:27.987174 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:27.987330 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:27.988155 (+ 825us) server_negotiation.cc:299] Negotiation successful
0808 03:58:27.988405 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260808 03:58:27.989720 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:27.966946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:33577)
0808 03:58:27.968101 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:27.968165 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:27.970592 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:27.971402 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:27.971415 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:27.972209 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:27.973107 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:27.973136 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:27.974255 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:27.974260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:27.976195 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:27.976222 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:27.977144 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:27.977151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:27.977465 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:27.978246 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:27.978279 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:27.981193 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:27.985419 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:27.985429 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:27.985445 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:27.985977 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:27.986445 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:27.986449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:27.986451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:27.986680 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:27.987297 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:27.987303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:27.987751 (+ 448us) client_negotiation.cc:769] Sending connection context
0808 03:58:27.988008 (+ 257us) client_negotiation.cc:240] Negotiation successful
0808 03:58:27.988483 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":552,"threads_started":1}
I20260808 03:58:27.991380 11505 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37713
I20260808 03:58:27.991753 11505 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:27.992537 11505 heartbeater.cc:511] Master 127.28.251.254:37713 requested a full tablet report, sending...
I20260808 03:58:27.994261 10932 ts_manager.cc:194] Registered new tserver with Master: fcb25bcf21b14c69b35daa3fc2237669 (127.28.251.194:36251)
I20260808 03:58:27.995450 10932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:33577
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:58:28.035244 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37713
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:28.401830 11517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:28.402148 11517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:28.402256 11517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:28.412276 11517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:28.412395 11517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:28.412520 11517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:58:28.422863 11517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:28.424314 11517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:28.425721 11517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:28.439823 11529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:28.439877 11534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:28.439929 11524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:28.440291 11264 heartbeater.cc:503] Master 127.28.251.254:37713 was elected leader, sending a full tablet report...
I20260808 03:58:28.440659 11517 server_base.cc:1034] running on GCE node
I20260808 03:58:28.441380 11517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:28.442627 11517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:28.443902 11517 hybrid_clock.cc:648] HybridClock initialized: now 1786161508443835 us; error 49 us; skew 500 ppm
I20260808 03:58:28.444278 11517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:28 dist-test-slave-pww0 krb5kdc[10860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161508, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:28.447628 11517 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:58:28.449174 11517 webserver.cc:460] Webserver started at http://127.28.251.195:44435/ using document root <none> and password file <none>
I20260808 03:58:28.449898 11517 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:28.450006 11517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:28.450299 11517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:28.453601 11517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "c23bcb9de53247dda1babd9397a3449f"
format_stamp: "Formatted at 2026-08-08 03:58:28 on dist-test-slave-pww0"
server_key: "3832edc11ca62ffab5de076e2816d9d1"
server_key_iv: "968e7abae5a25f6b7556c8c03a7b219a"
server_key_version: "encryptionkey@0"
I20260808 03:58:28.454458 11517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "c23bcb9de53247dda1babd9397a3449f"
format_stamp: "Formatted at 2026-08-08 03:58:28 on dist-test-slave-pww0"
server_key: "3832edc11ca62ffab5de076e2816d9d1"
server_key_iv: "968e7abae5a25f6b7556c8c03a7b219a"
server_key_version: "encryptionkey@0"
I20260808 03:58:28.460289 11517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 03:58:28.464354 11544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:28.466068 11517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 03:58:28.466315 11517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "c23bcb9de53247dda1babd9397a3449f"
format_stamp: "Formatted at 2026-08-08 03:58:28 on dist-test-slave-pww0"
server_key: "3832edc11ca62ffab5de076e2816d9d1"
server_key_iv: "968e7abae5a25f6b7556c8c03a7b219a"
server_key_version: "encryptionkey@0"
I20260808 03:58:28.466552 11517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:28.482887 11517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:28.488092 11517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:28.488451 11517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:28.489630 11517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:28.491379 11517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:28.491523 11517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 03:58:28.491668 11517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:28.491748 11517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:28.526520 11517 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:43529
I20260808 03:58:28.526562 11761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:43529 every 8 connection(s)
I20260808 03:58:28.528074 11517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:58:28.537320 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 11517
I20260808 03:58:28.537499 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:58:28.538120 29679 external_mini_cluster.cc:1428] Setting key 1218c7eb368c05d09ff42d44023cf3fb
Aug 08 03:58:28 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161508, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:28.550114 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:28.531109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:48139 (local address 127.28.251.254:37713)
0808 03:58:28.531703 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:28.531710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:28.532779 (+ 1069us) server_negotiation.cc:407] Connection header received
0808 03:58:28.534709 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:28.534714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:28.534807 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:28.534924 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:28.536698 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:28.537531 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:28.539763 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:28.540259 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:28.540380 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:28.544213 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:28.544221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:28.544226 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:28.544295 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:28.547040 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:28.547991 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:28.547996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:28.547998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:28.548130 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:28.548508 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:28.548511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:28.548513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:28.548786 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:28.548916 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:28.549624 (+ 708us) server_negotiation.cc:299] Negotiation successful
0808 03:58:28.549807 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":266,"threads_started":1}
I20260808 03:58:28.551132 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:28.531734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:48139)
0808 03:58:28.532571 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:28.532634 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:28.534391 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:28.535172 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:28.535185 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:28.535866 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:28.536497 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:28.536517 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:28.537741 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:28.537747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:28.539608 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:28.539625 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:28.540542 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:28.540551 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:28.540806 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:28.541570 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:28.541605 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:28.543955 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:28.547212 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:28.547222 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:28.547237 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:28.547869 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:28.548214 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:28.548217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:28.548219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:28.548424 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:28.548914 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:28.548920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:28.549280 (+ 360us) client_negotiation.cc:769] Sending connection context
0808 03:58:28.549529 (+ 249us) client_negotiation.cc:240] Negotiation successful
0808 03:58:28.549961 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":325,"threads_started":1}
I20260808 03:58:28.552932 11762 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37713
I20260808 03:58:28.553342 11762 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:28.554302 11762 heartbeater.cc:511] Master 127.28.251.254:37713 requested a full tablet report, sending...
I20260808 03:58:28.555819 10932 ts_manager.cc:194] Registered new tserver with Master: c23bcb9de53247dda1babd9397a3449f (127.28.251.195:43529)
I20260808 03:58:28.556679 10932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:48139
I20260808 03:58:28.566859 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab.
Entry for principal oryx/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab.
WARNING: no policy specified for oryx/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab.
Entry for principal oryx/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab.
WARNING: no policy specified for oryx/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab.
Entry for principal oryx/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab.
I20260808 03:58:28.753871 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 11028
Aug 08 03:58:28 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:58:28.770946 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:28.755059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.28.251.193:39109)
0808 03:58:28.755775 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:28.755784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:28.755813 (+ 29us) server_negotiation.cc:407] Connection header received
0808 03:58:28.755881 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:28.755888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:28.756104 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:28.756238 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:28.757638 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:28.758793 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:28.760185 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:28.760798 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:28.760915 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:28.764991 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:28.765008 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:28.765025 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:28.765075 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:28.768094 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:28.768801 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:28.768809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:28.768811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:28.768949 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:28.769304 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:28.769309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:28.769311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:28.769714 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:28.769970 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:28.770293 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 03:58:28.770619 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":383,"threads_started":1}
I20260808 03:58:28.772322 11254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:28.857183 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 11028
I20260808 03:58:28.883633 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 11272
Aug 08 03:58:28 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 03:58:28.900674 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:28.884671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34136 (local address 127.28.251.194:36251)
0808 03:58:28.885254 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:28.885262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:28.885289 (+ 27us) server_negotiation.cc:407] Connection header received
0808 03:58:28.885366 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:28.885373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:28.885592 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:28.885724 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:28.887001 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:28.888354 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:28.889662 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:28.890249 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:28.890385 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:28.894767 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:28.894784 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:28.894801 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:28.894846 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:28.897893 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:28.898630 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:28.898637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:28.898640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:28.898780 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:28.899085 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:28.899088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:28.899089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:28.899500 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:28.899775 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:28.900044 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 03:58:28.900271 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":276,"threads_started":1}
I20260808 03:58:28.902179 11497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:28.992275 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 11272
I20260808 03:58:29.013460 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 11517
Aug 08 03:58:29 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 03:58:29.031031 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:29.014523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.28.251.195:43529)
0808 03:58:29.015112 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:29.015119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:29.015152 (+ 33us) server_negotiation.cc:407] Connection header received
0808 03:58:29.015218 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:29.015225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:29.015574 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:29.015761 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:29.017248 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:29.018287 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:29.019641 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:29.020235 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:29.020363 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:29.025032 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:29.025051 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:29.025068 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:29.025114 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:29.027981 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:29.028750 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:29.028759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:29.028762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:29.028911 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:29.029310 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:29.029315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:29.029317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:29.029713 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:29.030048 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:29.030370 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 03:58:29.030650 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":275,"threads_started":1}
I20260808 03:58:29.032521 11755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:29.116962 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 11517
I20260808 03:58:29.150235 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 10883
I20260808 03:58:29.151707 10975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:29.258392 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 10883
I20260808 03:58:29.288242 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:37713
--webserver_interface=127.28.251.254
--webserver_port=39325
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:37713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:29.645249 11794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:29.645574 11794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:29.645720 11794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:29.653249 11794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:58:29.653378 11794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:29.653434 11794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:29.653515 11794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:58:29.653584 11794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:58:29.663981 11794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:37713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:37713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=39325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:29.665446 11794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:29.666836 11794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:29.679203 11801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:29.679124 11802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:29.679177 11810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:29.680042 11794 server_base.cc:1034] running on GCE node
I20260808 03:58:29.680928 11794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:29.682500 11794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:29.683722 11794 hybrid_clock.cc:648] HybridClock initialized: now 1786161509683662 us; error 51 us; skew 500 ppm
I20260808 03:58:29.684098 11794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:29 dist-test-slave-pww0 krb5kdc[10860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161509, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:29.691268 11794 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:58:29.693075 11794 webserver.cc:460] Webserver started at http://127.28.251.254:39325/ using document root <none> and password file <none>
I20260808 03:58:29.693885 11794 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:29.694044 11794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:29.700507 11794 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 03:58:29.703876 11820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:29.705456 11794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 03:58:29.705647 11794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "f5ead044f95f4a3c842fbd654241e37d"
format_stamp: "Formatted at 2026-08-08 03:58:26 on dist-test-slave-pww0"
server_key: "3c355e0df355c3d3bcae0a8ea8837f7d"
server_key_iv: "99b0089afbe04b8936b3094e41f2e57c"
server_key_version: "encryptionkey@0"
I20260808 03:58:29.706702 11794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:29.729982 11794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:29.734916 11794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:29.735216 11794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:29.754220 11794 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:37713
I20260808 03:58:29.754271 11907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:37713 every 8 connection(s)
I20260808 03:58:29.755899 11794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:58:29.758767 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 11794
I20260808 03:58:29.759697 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:39109
--local_ip_for_outbound_sockets=127.28.251.193
--tserver_master_addrs=127.28.251.254:37713
--webserver_port=46869
--webserver_interface=127.28.251.193
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.251.193
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 03:58:29.764117 11909 sys_catalog.cc:263] Verifying existing consensus state
I20260808 03:58:29.768054 11909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d: Bootstrap starting.
I20260808 03:58:29.794452 11909 log.cc:824] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:29.808073 11909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 03:58:29.808845 11909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d: Bootstrap complete.
I20260808 03:58:29.820919 11909 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } }
I20260808 03:58:29.821842 11909 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ead044f95f4a3c842fbd654241e37d, State: Initialized, Role: FOLLOWER
I20260808 03:58:29.822521 11909 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } }
I20260808 03:58:29.822764 11909 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:58:29.822883 11909 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:58:29.823051 11909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 1 FOLLOWER]: Advancing to term 2
I20260808 03:58:29.827174 11909 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } }
I20260808 03:58:29.827778 11909 leader_election.cc:302] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5ead044f95f4a3c842fbd654241e37d; no voters:
I20260808 03:58:29.828397 11909 leader_election.cc:288] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 03:58:29.828641 11915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 03:58:29.829643 11915 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [term 2 LEADER]: Becoming Leader. State: Replica: f5ead044f95f4a3c842fbd654241e37d, State: Running, Role: LEADER
I20260808 03:58:29.830540 11909 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:58:29.830386 11915 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } }
I20260808 03:58:29.834070 11917 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5ead044f95f4a3c842fbd654241e37d. Latest consensus state: current_term: 2 leader_uuid: "f5ead044f95f4a3c842fbd654241e37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } } }
I20260808 03:58:29.834542 11917 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:29.835131 11916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f5ead044f95f4a3c842fbd654241e37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5ead044f95f4a3c842fbd654241e37d" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37713 } } }
I20260808 03:58:29.835486 11916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:29.844977 11928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:58:29.851255 11928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:58:29.853116 11928 catalog_manager.cc:1259] Loaded cluster ID: 68e8fe1d675b428e9c6fb14368a28bd1
I20260808 03:58:29.853247 11928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:58:29.857899 11928 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:58:29.859961 11928 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f5ead044f95f4a3c842fbd654241e37d: Loaded TSK: 0
I20260808 03:58:29.861943 11928 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 03:58:30.169806 11912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:30.170149 11912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:30.170277 11912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:30.178166 11912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:30.178308 11912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:30.178458 11912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:58:30.189134 11912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.193.keytab
--principal=oryx/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:39109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=46869
--tserver_master_addrs=127.28.251.254:37713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:30.190659 11912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:30.192198 11912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:30.206627 11946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:30.206748 11947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:30.207508 11951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:30.212396 11912 server_base.cc:1034] running on GCE node
I20260808 03:58:30.213109 11912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:30.214309 11912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:30.215559 11912 hybrid_clock.cc:648] HybridClock initialized: now 1786161510215494 us; error 56 us; skew 500 ppm
I20260808 03:58:30.216037 11912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161510, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:30.219698 11912 init.cc:377] Logged in from keytab as oryx/127.28.251.193@KRBTEST.COM (short username oryx)
I20260808 03:58:30.221454 11912 webserver.cc:460] Webserver started at http://127.28.251.193:46869/ using document root <none> and password file <none>
I20260808 03:58:30.222189 11912 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:30.222342 11912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:30.228221 11912 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 03:58:30.231124 11961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:30.232612 11912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 03:58:30.232782 11912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "03aea75666404ddc8b33d67b65822c75"
format_stamp: "Formatted at 2026-08-08 03:58:27 on dist-test-slave-pww0"
server_key: "8ddbd3086708ddc9a24dd1b5d86f6b6a"
server_key_iv: "2a94c0b5d895ca20cdfad5c940954b66"
server_key_version: "encryptionkey@0"
I20260808 03:58:30.233808 11912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:30.258183 11912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:30.262605 11912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:30.262904 11912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:30.264037 11912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:30.265630 11912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:30.265743 11912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:30.265856 11912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:30.265929 11912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:30.299391 11912 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:39109
I20260808 03:58:30.299494 12132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:39109 every 8 connection(s)
I20260808 03:58:30.301011 11912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:58:30.309406 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 11912
I20260808 03:58:30.310138 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:36251
--local_ip_for_outbound_sockets=127.28.251.194
--tserver_master_addrs=127.28.251.254:37713
--webserver_port=35987
--webserver_interface=127.28.251.194
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.251.194
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:30.321053 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:30.304222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:39089)
0808 03:58:30.305091 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:30.305138 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:30.306921 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:30.308479 (+ 1558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:30.308488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:30.309066 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:30.309714 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.309737 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.312236 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.312241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.313652 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.313669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.315015 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.315020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.315226 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.315960 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:30.315992 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:30.319993 (+ 4001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":401,"threads_started":1}
W20260808 03:58:30.321708 12144 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:30.304093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:39089 (local address 127.28.251.254:37713)
0808 03:58:30.304886 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:30.304904 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:30.305278 (+ 374us) server_negotiation.cc:407] Connection header received
0808 03:58:30.307536 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:30.307552 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:30.307952 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:30.308279 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:30.309963 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.311998 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.313823 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.314777 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.314981 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.320502 (+ 5521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:39089: BlockingRecv error: recv got EOF from 127.28.251.193:39089 (error 108)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":270,"threads_started":1}
W20260808 03:58:30.321733 12133 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:37713 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.251.254:37713: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:30.330507 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:30.322378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:55743)
0808 03:58:30.322553 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:30.322571 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:30.322679 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:30.323058 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:30.323063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:30.323411 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:30.323609 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.323618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.324666 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.324671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.325587 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.325599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.326289 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.326296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.326474 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.327081 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:30.327098 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:30.330171 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 03:58:30.330720 12144 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:30.322590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:55743 (local address 127.28.251.254:37713)
0808 03:58:30.322779 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:30.322785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:30.322803 (+ 18us) server_negotiation.cc:407] Connection header received
0808 03:58:30.322859 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:30.322862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:30.322920 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:30.323004 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:30.323755 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.324515 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.325733 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.326136 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.326251 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.330469 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:55743: BlockingRecv error: recv got EOF from 127.28.251.193:55743 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:30.340282 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:30.331359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:44789)
0808 03:58:30.331542 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:30.331561 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:30.331662 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:30.332081 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:30.332085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:30.332374 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:30.332536 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.332544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.333972 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.333976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.334952 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.334965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.335922 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.335928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.336081 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.336760 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:30.336783 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:30.340005 (+ 3222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 03:58:30.340632 12133 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 03:58:30.341025 12144 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:30.331504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:44789 (local address 127.28.251.254:37713)
0808 03:58:30.331686 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:30.331692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:30.331715 (+ 23us) server_negotiation.cc:407] Connection header received
0808 03:58:30.331782 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:30.331787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:30.331937 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:30.332068 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:30.332684 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.333778 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.335112 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.335728 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.335880 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.340326 (+ 4446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:44789: BlockingRecv error: recv got EOF from 127.28.251.193:44789 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 03:58:30.677696 12147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:30.678013 12147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:30.678119 12147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:30.685678 12147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:30.685814 12147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:30.685941 12147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:58:30.696476 12147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.194.keytab
--principal=oryx/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:36251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=35987
--tserver_master_addrs=127.28.251.254:37713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:30.698004 12147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:30.699404 12147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:30.713402 12155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:30.713500 12160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:30.713474 12157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:30.714406 12147 server_base.cc:1034] running on GCE node
I20260808 03:58:30.715107 12147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:30.716292 12147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:30.717514 12147 hybrid_clock.cc:648] HybridClock initialized: now 1786161510717454 us; error 47 us; skew 500 ppm
I20260808 03:58:30.717878 12147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161510, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:30.721539 12147 init.cc:377] Logged in from keytab as oryx/127.28.251.194@KRBTEST.COM (short username oryx)
I20260808 03:58:30.723165 12147 webserver.cc:460] Webserver started at http://127.28.251.194:35987/ using document root <none> and password file <none>
I20260808 03:58:30.723975 12147 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:30.724135 12147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:30.730037 12147 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 03:58:30.733141 12178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:30.734601 12147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260808 03:58:30.734779 12147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "fcb25bcf21b14c69b35daa3fc2237669"
format_stamp: "Formatted at 2026-08-08 03:58:27 on dist-test-slave-pww0"
server_key: "fb968ac4f1ba4f46a0dae22a3578f44e"
server_key_iv: "3bd92c7c8a6d475aa479ef1f8fe84c7b"
server_key_version: "encryptionkey@0"
I20260808 03:58:30.735860 12147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:30.754917 12147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:30.760104 12147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:30.760401 12147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:30.761493 12147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:30.762938 12147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:30.763031 12147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:30.763145 12147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:30.763218 12147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:30.795686 12147 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:36251
I20260808 03:58:30.795770 12377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:36251 every 8 connection(s)
I20260808 03:58:30.797287 12147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:58:30.806161 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 12147
I20260808 03:58:30.806833 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:43529
--local_ip_for_outbound_sockets=127.28.251.195
--tserver_master_addrs=127.28.251.254:37713
--webserver_port=44435
--webserver_interface=127.28.251.195
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.251.195
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 03:58:30.814615 12144 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:30.800021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:46139 (local address 127.28.251.254:37713)
0808 03:58:30.800203 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:30.800208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:30.801469 (+ 1261us) server_negotiation.cc:407] Connection header received
0808 03:58:30.803406 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:30.803412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:30.803530 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:30.803623 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:30.805256 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.806139 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.807995 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.808654 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.808792 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.814195 (+ 5403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:46139: BlockingRecv error: recv got EOF from 127.28.251.194:46139 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 03:58:30.815119 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:30.800536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:46139)
0808 03:58:30.801298 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:30.801341 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:30.803104 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:30.803905 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:30.803914 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:30.804459 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:30.805067 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.805086 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.806378 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.806388 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.807831 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.807846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.808906 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.808911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.809161 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.809908 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:30.809941 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:30.813918 (+ 3977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":302,"threads_started":1}
W20260808 03:58:30.815882 12379 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:37713 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.251.254:37713: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:30.824892 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:30.816518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:39015)
0808 03:58:30.816695 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:30.816717 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:30.816869 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:30.817200 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:30.817205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:30.817615 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:30.817774 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.817783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.818856 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.818862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.819944 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.819962 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.820725 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.820732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.820929 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.821534 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:30.821554 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:30.824666 (+ 3112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 03:58:30.825256 12144 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:30.816643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:39015 (local address 127.28.251.254:37713)
0808 03:58:30.816813 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:30.816818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:30.816839 (+ 21us) server_negotiation.cc:407] Connection header received
0808 03:58:30.816994 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:30.817000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:30.817070 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:30.817151 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:30.817931 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.818703 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.820131 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.820566 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.820670 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.824978 (+ 4308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:39015: BlockingRecv error: recv got EOF from 127.28.251.194:39015 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:30 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:30.835016 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:30.825664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:46645)
0808 03:58:30.825860 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:30.825880 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:30.825988 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:30.826293 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:30.826297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:30.826679 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:30.826904 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.826916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.828094 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.828099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.829474 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:30.829492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.830270 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.830275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:30.830502 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.831146 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:30.831169 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:30.834671 (+ 3502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260808 03:58:30.835074 12144 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:30.825795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:46645 (local address 127.28.251.254:37713)
0808 03:58:30.825938 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:30.825942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:30.825959 (+ 17us) server_negotiation.cc:407] Connection header received
0808 03:58:30.826097 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:30.826101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:30.826160 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:30.826239 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:30.827086 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.827936 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.829654 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:30.830104 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:30.830236 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:30.834898 (+ 4662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:46645: BlockingRecv error: recv got EOF from 127.28.251.194:46645 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 03:58:30.835497 12379 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 03:58:31.181620 12383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:31.181972 12383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:31.182081 12383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:31.189790 12383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:31.189949 12383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:31.190076 12383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:58:31.200560 12383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/krb5kdc/oryx_127.28.251.195.keytab
--principal=oryx/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:43529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=44435
--tserver_master_addrs=127.28.251.254:37713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:31.202080 12383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:31.203639 12383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:31.218585 12394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:31.218537 12397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:31.218638 12399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:31.219578 12383 server_base.cc:1034] running on GCE node
I20260808 03:58:31.220469 12383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:31.221560 12383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:31.222775 12383 hybrid_clock.cc:648] HybridClock initialized: now 1786161511222709 us; error 62 us; skew 500 ppm
I20260808 03:58:31.223102 12383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:31 dist-test-slave-pww0 krb5kdc[10860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161511, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:31.226717 12383 init.cc:377] Logged in from keytab as oryx/127.28.251.195@KRBTEST.COM (short username oryx)
I20260808 03:58:31.228542 12383 webserver.cc:460] Webserver started at http://127.28.251.195:44435/ using document root <none> and password file <none>
I20260808 03:58:31.229351 12383 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:31.229499 12383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:31.235827 12383 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 03:58:31.238941 12409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:31.240753 12383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:31.240960 12383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "c23bcb9de53247dda1babd9397a3449f"
format_stamp: "Formatted at 2026-08-08 03:58:28 on dist-test-slave-pww0"
server_key: "3832edc11ca62ffab5de076e2816d9d1"
server_key_iv: "968e7abae5a25f6b7556c8c03a7b219a"
server_key_version: "encryptionkey@0"
I20260808 03:58:31.242045 12383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:31.270941 12383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:31.275781 12383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:31.276038 12383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:31.277072 12383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:31.278587 12383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:31.278688 12383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:31.278774 12383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:31.278829 12383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:31.311827 12383 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:43529
I20260808 03:58:31.311889 12593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:43529 every 8 connection(s)
I20260808 03:58:31.313486 12383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:58:31.314546 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 12383
Aug 08 03:58:31 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:31 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 03:58:31.332381 12597 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:31.316692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:48359 (local address 127.28.251.254:37713)
0808 03:58:31.319399 (+ 2707us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:31.319404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:31.319420 (+ 16us) server_negotiation.cc:407] Connection header received
0808 03:58:31.320693 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:31.320699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:31.320786 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:31.320885 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:31.322823 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.323679 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.325353 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.325871 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.325972 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:31.332028 (+ 6056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:48359: BlockingRecv error: recv got EOF from 127.28.251.195:48359 (error 108)
Metrics: {"server-negotiator.queue_time_us":2615,"thread_start_us":2453,"threads_started":1}
I20260808 03:58:31.333053 12598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:31.317191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:48359)
0808 03:58:31.318008 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:31.318063 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:31.320377 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:31.321186 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:31.321195 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:31.321778 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:31.322642 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:31.322670 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.323843 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.323849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:31.325172 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:31.325185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.326115 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.326123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:31.326408 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:31.327069 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:31.327105 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:31.331675 (+ 4570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":333,"threads_started":1}
W20260808 03:58:31.333725 12594 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:37713 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.251.254:37713: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
I20260808 03:58:31.333853 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:31.315703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40964 (local address 127.28.251.254:37713)
0808 03:58:31.315895 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:31.315899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:31.316499 (+ 600us) server_negotiation.cc:407] Connection header received
0808 03:58:31.316714 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:31.316720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:31.316808 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:31.316925 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:31.319086 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.320336 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.321790 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.322395 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.322538 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:31.325446 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:31.325454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:31.325471 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:31.325515 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:31.329376 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:31.330321 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:31.330330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:31.330337 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:31.330547 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:31.331052 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:31.331056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:31.331058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:31.331714 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:31.332067 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:31.332700 (+ 633us) server_negotiation.cc:299] Negotiation successful
0808 03:58:31.333056 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 03:58:31 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:31 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:31.343107 12598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:31.334367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:37999)
0808 03:58:31.334569 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:31.334591 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:31.334715 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:31.335036 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:31.335039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:31.335356 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:31.335550 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:31.335564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.336892 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.336896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:31.338000 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:31.338015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.338796 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.338801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:31.338979 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:31.339556 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:31.339575 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:31.342812 (+ 3237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260808 03:58:31.343679 12597 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:31.334597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:37999 (local address 127.28.251.254:37713)
0808 03:58:31.334775 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:31.334780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:31.334798 (+ 18us) server_negotiation.cc:407] Connection header received
0808 03:58:31.334848 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:31.334853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:31.334921 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:31.335020 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:31.335687 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.336743 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.338201 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.338665 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.338774 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:31.343030 (+ 4256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:37999: BlockingRecv error: recv got EOF from 127.28.251.195:37999 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 03:58:31 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:31 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:31 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:31.351173 12600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:31.341257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:33671)
0808 03:58:31.341804 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:31.341819 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:31.341954 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:31.342386 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:31.342390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:31.342815 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:31.343018 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:31.343026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.344341 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.344347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:31.345641 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:31.345655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.346487 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.346494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:31.346649 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:31.347279 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:31.347306 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:31.350862 (+ 3556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":267,"threads_started":1}
W20260808 03:58:31.351374 12144 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:31.341313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:33671 (local address 127.28.251.254:37713)
0808 03:58:31.341604 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:31.341610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:31.341905 (+ 295us) server_negotiation.cc:407] Connection header received
0808 03:58:31.342105 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:31.342129 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:31.342240 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:31.342348 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:31.343166 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.344195 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.345776 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.346301 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.346412 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:31.351141 (+ 4729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:33671: BlockingRecv error: recv got EOF from 127.28.251.193:33671 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Aug 08 03:58:31 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:31.352447 12598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:31.343875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:37815)
0808 03:58:31.344025 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:31.344041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:31.344137 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:31.344606 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:31.344611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:31.345006 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:31.345198 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:31.345210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.346330 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.346335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:31.347581 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:31.347598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.348367 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.348374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:31.348589 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:31.349236 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:31.349260 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:31.352233 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 03:58:31.352602 12597 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:31.344215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:37815 (local address 127.28.251.254:37713)
0808 03:58:31.344347 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:31.344350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:31.344364 (+ 14us) server_negotiation.cc:407] Connection header received
0808 03:58:31.344400 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:31.344403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:31.344467 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:31.344537 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:31.345346 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.346170 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.347723 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.348194 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.348333 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:31.352400 (+ 4067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:37815: BlockingRecv error: recv got EOF from 127.28.251.195:37815 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 03:58:31.352784 12594 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 03:58:31 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:31 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:31.845547 12601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:31.836192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:46349)
0808 03:58:31.836733 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:31.836750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:31.836892 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:31.837229 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:31.837232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:31.837755 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:31.837988 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:31.837998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.839207 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.839211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:31.840343 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:31.840361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.841337 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.841344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:31.841526 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:31.842221 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:31.842241 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:31.845290 (+ 3049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":276,"threads_started":1}
W20260808 03:58:31.845731 12597 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:31.836318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:46349 (local address 127.28.251.254:37713)
0808 03:58:31.836497 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:31.836502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:31.836863 (+ 361us) server_negotiation.cc:407] Connection header received
0808 03:58:31.837019 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:31.837023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:31.837094 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:31.837191 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:31.838184 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.839057 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.840549 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:31.841121 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:31.841262 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:31.845530 (+ 4268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:46349: BlockingRecv error: recv got EOF from 127.28.251.194:46349 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 03:58:32 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:32 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:32 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:32.362362 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:32.352316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:42351)
0808 03:58:32.353216 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:32.353238 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:32.353423 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:32.353762 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:32.353768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:32.354373 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:32.354624 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:32.354638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.356114 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.356120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:32.357061 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:32.357074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.357792 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.357799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:32.357970 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:32.358650 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:32.358675 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:32.362002 (+ 3327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":317,"threads_started":1}
Aug 08 03:58:32 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 03:58:32.362880 12605 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:32.352507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:42351 (local address 127.28.251.254:37713)
0808 03:58:32.353198 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:32.353204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:32.353479 (+ 275us) server_negotiation.cc:407] Connection header received
0808 03:58:32.353541 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:32.353546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:32.353614 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:32.353709 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:32.354787 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.355969 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.357220 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.357634 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.357747 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:32.362199 (+ 4452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:42351: BlockingRecv error: recv got EOF from 127.28.251.193:42351 (error 108)
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":421,"threads_started":1}
I20260808 03:58:32.363751 12610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:32.353444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:52953)
0808 03:58:32.354967 (+ 1523us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:32.354985 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:32.355110 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:32.355508 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:32.355513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:32.355996 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:32.356193 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:32.356203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.357295 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.357300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:32.358327 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:32.358343 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.359293 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.359298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:32.359438 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:32.360001 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:32.360020 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:32.363405 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1414,"thread_start_us":1102,"threads_started":1}
W20260808 03:58:32.364002 12607 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:32.353444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:52953 (local address 127.28.251.254:37713)
0808 03:58:32.354259 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:32.354265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:32.355071 (+ 806us) server_negotiation.cc:407] Connection header received
0808 03:58:32.355222 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:32.355228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:32.355299 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:32.355438 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:32.356362 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.357145 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.358483 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.359118 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.359249 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:32.363745 (+ 4496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:52953: BlockingRecv error: recv got EOF from 127.28.251.195:52953 (error 108)
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":534,"threads_started":1}
Aug 08 03:58:32 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:32 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:32.855590 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:32.846625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:56427)
0808 03:58:32.847142 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:32.847163 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:32.847289 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:32.847676 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:32.847680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:32.848099 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:32.848298 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:32.848308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.849350 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.849355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:32.850339 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:32.850366 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.851001 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.851005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:32.851135 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:32.851819 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:32.851837 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:32.855248 (+ 3411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":273,"threads_started":1}
W20260808 03:58:32.855671 12607 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:32.846712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:56427 (local address 127.28.251.254:37713)
0808 03:58:32.846969 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:32.846975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:32.847237 (+ 262us) server_negotiation.cc:407] Connection header received
0808 03:58:32.847388 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:32.847392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:32.847487 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:32.847577 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:32.848449 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.849205 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.850473 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:32.850884 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:32.850975 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:32.855439 (+ 4464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:56427: BlockingRecv error: recv got EOF from 127.28.251.194:56427 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 03:58:33 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:33 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:33 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:33 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:33.374711 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:33.363833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:40203)
0808 03:58:33.364476 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:33.364501 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:33.364664 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:33.365284 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:33.365291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:33.365850 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:33.366066 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:33.366078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.367626 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.367632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:33.368908 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:33.368925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.369957 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.369962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:33.370126 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:33.370887 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:33.370914 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:33.374304 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":280,"threads_started":1}
W20260808 03:58:33.374869 12615 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:33.364118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:40203 (local address 127.28.251.254:37713)
0808 03:58:33.364715 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:33.364720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:33.364740 (+ 20us) server_negotiation.cc:407] Connection header received
0808 03:58:33.364800 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:33.364804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:33.364898 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:33.365020 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:33.366233 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.367438 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.369055 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.369750 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.369888 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:33.374566 (+ 4678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:40203: BlockingRecv error: recv got EOF from 127.28.251.193:40203 (error 108)
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":485,"thread_start_us":353,"threads_started":1}
I20260808 03:58:33.375038 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:33.364755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:51659)
0808 03:58:33.365780 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:33.365802 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:33.365947 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:33.366475 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:33.366480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:33.367027 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:33.367272 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:33.367286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.368672 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.368676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:33.369985 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:33.370004 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.370958 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.370962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:33.371088 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:33.371709 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:33.371728 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:33.374718 (+ 2990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":777,"threads_started":1}
W20260808 03:58:33.375571 12618 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:33.365003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:51659 (local address 127.28.251.254:37713)
0808 03:58:33.366145 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:33.366149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:33.366163 (+ 14us) server_negotiation.cc:407] Connection header received
0808 03:58:33.366202 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:33.366206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:33.366256 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:33.366389 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:33.367501 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.368474 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.370153 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.370767 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.370902 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:33.374972 (+ 4070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:51659: BlockingRecv error: recv got EOF from 127.28.251.195:51659 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1046,"thread_start_us":875,"threads_started":1}
Aug 08 03:58:33 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:33 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:33.864962 12622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:33.856585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:49117)
0808 03:58:33.857223 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:33.857243 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:33.857385 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:33.857745 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:33.857749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:33.858133 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:33.858323 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:33.858332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.859310 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.859314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:33.860291 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:33.860304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.860960 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.860964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:33.861095 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:33.861774 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:33.861793 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:33.864681 (+ 2888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":350,"threads_started":1}
W20260808 03:58:33.865106 12618 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:33.856668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:49117 (local address 127.28.251.254:37713)
0808 03:58:33.856877 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:33.856883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:33.857343 (+ 460us) server_negotiation.cc:407] Connection header received
0808 03:58:33.857493 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:33.857497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:33.857565 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:33.857648 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:33.858452 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.859174 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.860413 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:33.860832 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:33.860927 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:33.864900 (+ 3973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:49117: BlockingRecv error: recv got EOF from 127.28.251.194:49117 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 03:58:34 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:34 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:34 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:34 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:34.387252 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:34.376095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:40105)
0808 03:58:34.377276 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:34.377298 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:34.377476 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:34.377916 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:34.377921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:34.378396 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:34.378609 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:34.378623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.380250 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.380256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:34.381792 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:34.381809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.382973 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.382979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:34.383159 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:34.383738 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:34.383760 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:34.386992 (+ 3232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":941,"threads_started":1}
W20260808 03:58:34.387658 12627 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:34.376190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:40105 (local address 127.28.251.254:37713)
0808 03:58:34.377093 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:34.377099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:34.377412 (+ 313us) server_negotiation.cc:407] Connection header received
0808 03:58:34.377619 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:34.377625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:34.377709 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:34.377858 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:34.378771 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.380110 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.381997 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.382648 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.382767 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:34.387313 (+ 4546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:40105: BlockingRecv error: recv got EOF from 127.28.251.193:40105 (error 108)
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":808,"spinlock_wait_cycles":3968,"thread_start_us":617,"threads_started":1}
I20260808 03:58:34.387713 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:34.376082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:33409)
0808 03:58:34.377492 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:34.377510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:34.377632 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:34.377932 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:34.377937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:34.378425 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:34.378620 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:34.378633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.380251 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.380256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:34.381773 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:34.381792 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.382751 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.382757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:34.382944 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:34.383638 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:34.383665 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:34.387281 (+ 3616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":1129,"threads_started":1}
W20260808 03:58:34.388000 12626 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:34.376182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:33409 (local address 127.28.251.254:37713)
0808 03:58:34.376705 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:34.376713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:34.377587 (+ 874us) server_negotiation.cc:407] Connection header received
0808 03:58:34.377736 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:34.377741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:34.377795 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:34.377886 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:34.378771 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.380105 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.381918 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.382578 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.382707 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:34.387619 (+ 4912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:33409: BlockingRecv error: recv got EOF from 127.28.251.195:33409 (error 108)
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":384,"thread_start_us":257,"threads_started":1}
Aug 08 03:58:34 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:34 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:34.880302 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:34.866070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:58911)
0808 03:58:34.866698 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:34.866720 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:34.866890 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:34.867320 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:34.867325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:34.867883 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:34.868098 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:34.868114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.869188 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.869195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:34.870199 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:34.870214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.870918 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.870925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:34.871120 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:34.876571 (+ 5451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:34.876594 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:34.879975 (+ 3381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":363,"threads_started":1}
W20260808 03:58:34.880540 12626 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:34.866195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:58911 (local address 127.28.251.254:37713)
0808 03:58:34.866410 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:34.866416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:34.866847 (+ 431us) server_negotiation.cc:407] Connection header received
0808 03:58:34.867032 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:34.867037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:34.867120 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:34.867226 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:34.868251 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.869031 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.870366 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:34.870763 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:34.870881 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:34.880294 (+ 9413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:58911: BlockingRecv error: recv got EOF from 127.28.251.194:58911 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 03:58:35 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:35 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:35 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:35 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:35.399238 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:35.388831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:41697)
0808 03:58:35.389433 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:35.389455 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:35.389593 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:35.390561 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:35.390566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:35.391000 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:35.391158 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:35.391168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.392566 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.392572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:35.394004 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:35.394017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.394918 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.394925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:35.395092 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:35.395738 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:35.395761 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:35.398908 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":357,"threads_started":1}
W20260808 03:58:35.399479 12637 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:35.389676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:41697 (local address 127.28.251.254:37713)
0808 03:58:35.390215 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:35.390219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:35.390236 (+ 17us) server_negotiation.cc:407] Connection header received
0808 03:58:35.390291 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:35.390295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:35.390367 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:35.390462 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:35.391323 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.392439 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.394131 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.394769 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.394873 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:35.399153 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:41697: BlockingRecv error: recv got EOF from 127.28.251.195:41697 (error 108)
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":281,"threads_started":1}
I20260808 03:58:35.399748 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:35.388558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:35535)
0808 03:58:35.389263 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:35.389292 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:35.389472 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:35.390039 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:35.390045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:35.390571 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:35.390803 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:35.390816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.392293 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.392298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:35.393702 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:35.393721 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.394531 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.394537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:35.394708 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:35.395574 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:35.395600 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:35.399211 (+ 3611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":293,"threads_started":1}
W20260808 03:58:35.399770 12632 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:35.388699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:35535 (local address 127.28.251.254:37713)
0808 03:58:35.389726 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:35.389731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:35.389747 (+ 16us) server_negotiation.cc:407] Connection header received
0808 03:58:35.389798 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:35.389802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:35.389866 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:35.389969 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:35.390994 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.392157 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.393858 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.394374 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.394514 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:35.399506 (+ 4992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:35535: BlockingRecv error: recv got EOF from 127.28.251.193:35535 (error 108)
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":751,"threads_started":1}
Aug 08 03:58:35 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:35 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:35.890090 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:35.881386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:44015)
0808 03:58:35.881966 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:35.881983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:35.882103 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:35.882416 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:35.882420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:35.882845 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:35.883014 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:35.883022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.884138 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.884142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:35.885111 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:35.885124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.885865 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.885869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:35.886003 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:35.886660 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:35.886680 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:35.889816 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":278,"threads_started":1}
W20260808 03:58:35.890290 12632 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:35.881645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:44015 (local address 127.28.251.254:37713)
0808 03:58:35.881822 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:35.881827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:35.882070 (+ 243us) server_negotiation.cc:407] Connection header received
0808 03:58:35.882202 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:35.882206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:35.882273 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:35.882375 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:35.883174 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.884018 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.885269 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:35.885711 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:35.885816 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:35.890056 (+ 4240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:44015: BlockingRecv error: recv got EOF from 127.28.251.194:44015 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 03:58:36 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:36 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:36 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:36.410076 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:36.400527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:43955)
0808 03:58:36.401120 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:36.401153 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:36.401303 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:36.401764 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:36.401768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:36.402215 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:36.402394 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:36.402404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.403721 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.403725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:36.404692 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:36.404706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.405440 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.405446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:36.405613 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:36.406305 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:36.406325 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:36.409790 (+ 3465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":279,"threads_started":1}
W20260808 03:58:36.410360 12643 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:36.400655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:43955 (local address 127.28.251.254:37713)
0808 03:58:36.401381 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:36.401386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:36.401405 (+ 19us) server_negotiation.cc:407] Connection header received
0808 03:58:36.401474 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:36.401479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:36.401565 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:36.401728 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:36.402557 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.403582 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.404842 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.405276 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.405388 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:36.410050 (+ 4662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:43955: BlockingRecv error: recv got EOF from 127.28.251.195:43955 (error 108)
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":469,"threads_started":1}
Aug 08 03:58:36 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:36.411521 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:36.400790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:48957)
0808 03:58:36.401631 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:36.401652 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:36.401789 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:36.403058 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:36.403062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:36.403588 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:36.403764 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:36.403775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.404929 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.404934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:36.405971 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:36.405990 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.406995 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.407003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:36.407352 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:36.408157 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:36.408183 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:36.411186 (+ 3003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":593,"threads_started":1}
W20260808 03:58:36.411715 12646 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:36.401075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:48957 (local address 127.28.251.254:37713)
0808 03:58:36.402693 (+ 1618us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:36.402699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:36.402714 (+ 15us) server_negotiation.cc:407] Connection header received
0808 03:58:36.402765 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:36.402769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:36.402853 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:36.402974 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:36.403929 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.404773 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.406194 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.406769 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.406898 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:36.411426 (+ 4528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:48957: BlockingRecv error: recv got EOF from 127.28.251.193:48957 (error 108)
Metrics: {"server-negotiator.queue_time_us":1490,"thread_start_us":551,"threads_started":1}
Aug 08 03:58:36 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:36 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:36.899881 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:36.891221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:49929)
0808 03:58:36.891819 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:36.891835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:36.891978 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:36.892273 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:36.892276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:36.892753 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:36.892996 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:36.893009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.894329 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.894336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:36.895431 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:36.895446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.896143 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.896147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:36.896281 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:36.896833 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:36.896856 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:36.899637 (+ 2781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":337,"threads_started":1}
W20260808 03:58:36.900074 12646 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:36.891298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:49929 (local address 127.28.251.254:37713)
0808 03:58:36.891555 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:36.891562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:36.891930 (+ 368us) server_negotiation.cc:407] Connection header received
0808 03:58:36.892081 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:36.892087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:36.892159 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:36.892242 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:36.893132 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.894172 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.895599 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:36.896011 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:36.896106 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:36.899848 (+ 3742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:49929: BlockingRecv error: recv got EOF from 127.28.251.194:49929 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 03:58:37 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:37 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:37 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:37 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:37.421648 12648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:37.411276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:46519)
0808 03:58:37.411869 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:37.411889 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:37.412036 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:37.412677 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:37.412682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:37.413251 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:37.413469 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:37.413484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.414876 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.414881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:37.416227 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:37.416242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.417150 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.417154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:37.417280 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:37.417877 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:37.417904 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:37.421396 (+ 3492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":445,"threads_started":1}
W20260808 03:58:37.421976 12650 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:37.411676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:46519 (local address 127.28.251.254:37713)
0808 03:58:37.412317 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:37.412322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:37.412342 (+ 20us) server_negotiation.cc:407] Connection header received
0808 03:58:37.412405 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:37.412410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:37.412489 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:37.412599 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:37.413623 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.414700 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.416375 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.416998 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.417131 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:37.421687 (+ 4556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:46519: BlockingRecv error: recv got EOF from 127.28.251.195:46519 (error 108)
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":403,"threads_started":1}
I20260808 03:58:37.422371 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:37.412476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:36579)
0808 03:58:37.413174 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:37.413196 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:37.413310 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:37.413766 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:37.413771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:37.414304 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:37.414516 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:37.414529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.416042 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.416047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:37.417414 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:37.417429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.418300 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.418305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:37.418423 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:37.418937 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:37.418954 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:37.422030 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":308,"threads_started":1}
W20260808 03:58:37.422557 12651 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:37.412513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:36579 (local address 127.28.251.254:37713)
0808 03:58:37.413438 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:37.413444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:37.413483 (+ 39us) server_negotiation.cc:407] Connection header received
0808 03:58:37.413533 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:37.413537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:37.413607 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:37.413701 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:37.414665 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.415886 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.417567 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.418130 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.418258 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:37.422310 (+ 4052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:36579: BlockingRecv error: recv got EOF from 127.28.251.193:36579 (error 108)
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":674,"threads_started":1}
Aug 08 03:58:37 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:37 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:37.909920 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:37.900873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:43999)
0808 03:58:37.901513 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:37.901531 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:37.901660 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:37.901992 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:37.901995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:37.902439 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:37.902624 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:37.902634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.904035 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.904039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:37.905174 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:37.905191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.905908 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.905913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:37.906041 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:37.906603 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:37.906619 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:37.909536 (+ 2917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":280,"threads_started":1}
W20260808 03:58:37.910009 12651 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:37.901047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:43999 (local address 127.28.251.254:37713)
0808 03:58:37.901287 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:37.901292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:37.901602 (+ 310us) server_negotiation.cc:407] Connection header received
0808 03:58:37.901778 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:37.901782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:37.901868 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:37.901948 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:37.902808 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.903860 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.905315 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:37.905748 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:37.905852 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:37.909783 (+ 3931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:43999: BlockingRecv error: recv got EOF from 127.28.251.194:43999 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Aug 08 03:58:38 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:38 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:38 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:38 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:38.434527 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:38.422680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:56219)
0808 03:58:38.423399 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:38.423419 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:38.424125 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:38.424560 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:38.424565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:38.425106 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:38.425335 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:38.425348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.427028 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.427035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:38.428708 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:38.428731 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.429730 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.429735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:38.429893 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:38.430641 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:38.430662 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:38.434250 (+ 3588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":309,"threads_started":1}
W20260808 03:58:38.434949 12655 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:38.422764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:56219 (local address 127.28.251.254:37713)
0808 03:58:38.423299 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:38.423305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:38.423531 (+ 226us) server_negotiation.cc:407] Connection header received
0808 03:58:38.424274 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:38.424278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:38.424387 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:38.424506 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:38.425475 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.426806 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.428911 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.429548 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.430036 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:38.434571 (+ 4535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:56219: BlockingRecv error: recv got EOF from 127.28.251.195:56219 (error 108)
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":404,"spinlock_wait_cycles":11520,"thread_start_us":731,"threads_started":1}
I20260808 03:58:38.434947 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:38.423334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:37193)
0808 03:58:38.423918 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:38.423939 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:38.424078 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:38.424560 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:38.424565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:38.425095 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:38.425310 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:38.425323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.427037 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.427043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:38.428699 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:38.429002 (+ 303us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.430263 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.430275 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:38.430444 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:38.431233 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:38.431256 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:38.434572 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":281,"threads_started":1}
W20260808 03:58:38.435500 12659 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:38.423626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:37193 (local address 127.28.251.254:37713)
0808 03:58:38.424185 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:38.424191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:38.424210 (+ 19us) server_negotiation.cc:407] Connection header received
0808 03:58:38.424271 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:38.424276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:38.424388 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:38.424506 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:38.425475 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.426850 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.429128 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.429794 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.429909 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:38.434857 (+ 4948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:37193: BlockingRecv error: recv got EOF from 127.28.251.193:37193 (error 108)
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":450,"spinlock_wait_cycles":4352,"thread_start_us":300,"threads_started":1}
Aug 08 03:58:38 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:38 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:38.919668 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:38.910957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:36165)
0808 03:58:38.911559 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:38.911577 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:38.911720 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:38.912074 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:38.912080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:38.912540 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:38.912704 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:38.912714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.913941 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.913947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:38.914902 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:38.914917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.915700 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.915708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:38.915866 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:38.916469 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:38.916487 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:38.919309 (+ 2822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":279,"threads_started":1}
W20260808 03:58:38.919814 12659 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:38.911017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:36165 (local address 127.28.251.254:37713)
0808 03:58:38.911196 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:38.911200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:38.911661 (+ 461us) server_negotiation.cc:407] Connection header received
0808 03:58:38.911854 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:38.911858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:38.911924 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:38.912005 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:38.912864 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.913782 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.915052 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:38.915495 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:38.915622 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:38.919569 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:36165: BlockingRecv error: recv got EOF from 127.28.251.194:36165 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 03:58:39 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:39 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:39 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:39 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:39.458190 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:39.435984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:37263)
0808 03:58:39.448879 (+ 12895us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:39.448899 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:39.449027 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:39.449642 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:39.449645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:39.450084 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:39.450317 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:39.450332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.451655 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.451660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:39.452733 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:39.452750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.453632 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.453636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:39.453784 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:39.454387 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:39.454407 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:39.457911 (+ 3504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12760,"thread_start_us":12615,"threads_started":1}
I20260808 03:58:39.458485 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:39.435876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:40399)
0808 03:58:39.448488 (+ 12612us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:39.448514 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:39.448693 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:39.449278 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:39.449284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:39.449868 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:39.450048 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:39.450059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.451349 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.451355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:39.452439 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:39.452457 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.453376 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.453383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:39.453556 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:39.454593 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:39.454613 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:39.458183 (+ 3570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12454,"thread_start_us":12315,"threads_started":1}
W20260808 03:58:39.458518 12666 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:39.436005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:37263 (local address 127.28.251.254:37713)
0808 03:58:39.449304 (+ 13299us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:39.449310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:39.449336 (+ 26us) server_negotiation.cc:407] Connection header received
0808 03:58:39.449387 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:39.449392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:39.449458 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:39.449562 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:39.450521 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.451509 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.452901 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.453463 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.453587 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:39.458172 (+ 4585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:37263: BlockingRecv error: recv got EOF from 127.28.251.193:37263 (error 108)
Metrics: {"server-negotiator.queue_time_us":13172,"thread_start_us":13013,"threads_started":1}
W20260808 03:58:39.459043 12664 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:39.435954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:40399 (local address 127.28.251.254:37713)
0808 03:58:39.448422 (+ 12468us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:39.448430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:39.448859 (+ 429us) server_negotiation.cc:407] Connection header received
0808 03:58:39.448943 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:39.448949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:39.449047 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:39.449187 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:39.450210 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.451125 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.452610 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.453165 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.453274 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:39.458469 (+ 5195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:40399: BlockingRecv error: recv got EOF from 127.28.251.195:40399 (error 108)
Metrics: {"server-negotiator.queue_time_us":12281,"thread_start_us":12830,"threads_started":1}
Aug 08 03:58:39 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:39 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:39.932585 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:39.920776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:37433)
0808 03:58:39.921328 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:39.921347 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:39.921486 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:39.921778 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:39.921781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:39.922227 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:39.922416 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:39.922427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.923494 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.923500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:39.924516 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:39.924531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.925254 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.925259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:39.925399 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:39.928929 (+ 3530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:39.928950 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:39.932229 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":278,"threads_started":1}
W20260808 03:58:39.932724 12664 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:39.920860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:37433 (local address 127.28.251.254:37713)
0808 03:58:39.921103 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:39.921109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:39.921430 (+ 321us) server_negotiation.cc:407] Connection header received
0808 03:58:39.921589 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:39.921593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:39.921660 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:39.921742 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:39.922542 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.923296 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.924664 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:39.925083 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:39.925183 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:39.932483 (+ 7300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:37433: BlockingRecv error: recv got EOF from 127.28.251.194:37433 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 03:58:40 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:40 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:40 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:40 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:40.483237 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:40.459436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:50223)
0808 03:58:40.474344 (+ 14908us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:40.474366 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:40.474547 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:40.474946 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:40.474951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:40.475387 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:40.475579 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:40.475589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.476719 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.476723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:40.477686 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:40.477699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.478558 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.478565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:40.478734 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:40.479381 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:40.479401 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:40.482897 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14787,"thread_start_us":15066,"threads_started":1}
I20260808 03:58:40.483259 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:40.459609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:46127)
0808 03:58:40.474272 (+ 14663us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:40.474291 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:40.474432 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:40.474823 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:40.474828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:40.475252 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:40.475442 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:40.475479 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.476586 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.476591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:40.477566 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:40.477580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.478406 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.478412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:40.478590 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:40.479202 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:40.479228 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:40.482900 (+ 3672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14548,"thread_start_us":14388,"threads_started":1}
W20260808 03:58:40.483477 12675 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:40.459600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:46127 (local address 127.28.251.254:37713)
0808 03:58:40.473987 (+ 14387us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:40.473994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:40.474405 (+ 411us) server_negotiation.cc:407] Connection header received
0808 03:58:40.474546 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:40.474551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:40.474629 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:40.474736 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:40.475620 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.476442 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.477713 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.478228 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.478346 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:40.483178 (+ 4832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:46127: BlockingRecv error: recv got EOF from 127.28.251.195:46127 (error 108)
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":14249,"thread_start_us":13794,"threads_started":1}
W20260808 03:58:40.483579 12676 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:40.459540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:50223 (local address 127.28.251.254:37713)
0808 03:58:40.473687 (+ 14147us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:40.473695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:40.474500 (+ 805us) server_negotiation.cc:407] Connection header received
0808 03:58:40.474683 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:40.474688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:40.474780 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:40.474883 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:40.475760 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.476581 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.477844 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.478392 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.478510 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:40.483215 (+ 4705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:50223: BlockingRecv error: recv got EOF from 127.28.251.193:50223 (error 108)
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":13976,"thread_start_us":14172,"threads_started":1}
Aug 08 03:58:40 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:40 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:40.943137 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:40.933739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:41681)
0808 03:58:40.934304 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:40.934324 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:40.934500 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:40.934917 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:40.934922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:40.935398 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:40.935630 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:40.935645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.937033 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.937038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:40.938140 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:40.938156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.938991 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.938997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:40.939133 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:40.939816 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:40.939839 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:40.942886 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":319,"threads_started":1}
W20260808 03:58:40.943373 12675 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:40.933837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:41681 (local address 127.28.251.254:37713)
0808 03:58:40.934059 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:40.934067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:40.934423 (+ 356us) server_negotiation.cc:407] Connection header received
0808 03:58:40.934652 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:40.934657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:40.934741 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:40.934852 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:40.935803 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.936852 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.938339 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:40.938782 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:40.938942 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:40.943143 (+ 4201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:41681: BlockingRecv error: recv got EOF from 127.28.251.194:41681 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 03:58:41 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:41 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:41 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:41 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:41.502274 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:41.484525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:43089)
0808 03:58:41.492426 (+ 7901us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:41.492443 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:41.492588 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:41.493063 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:41.493069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:41.493573 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:41.493771 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:41.493782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.495122 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.495129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:41.496650 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:41.496666 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.497693 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.497700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:41.497902 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:41.498594 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:41.498618 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:41.501902 (+ 3284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":7777,"thread_start_us":7668,"threads_started":1}
I20260808 03:58:41.502494 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:41.484525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:57657)
0808 03:58:41.491284 (+ 6759us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:41.491316 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:41.491516 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:41.492590 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:41.492596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:41.493159 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:41.493381 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:41.493392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.494932 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.494938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:41.496466 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:41.496487 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.497596 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.497603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:41.497804 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:41.498547 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:41.498574 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:41.502138 (+ 3564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6573,"thread_start_us":6403,"threads_started":1}
W20260808 03:58:41.502490 12682 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:41.484719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:43089 (local address 127.28.251.254:37713)
0808 03:58:41.491763 (+ 7044us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:41.491771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:41.492542 (+ 771us) server_negotiation.cc:407] Connection header received
0808 03:58:41.492826 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:41.492830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:41.492894 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:41.492984 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:41.493928 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.494997 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.496770 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.497509 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.497623 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:41.502185 (+ 4562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:43089: BlockingRecv error: recv got EOF from 127.28.251.195:43089 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":6881,"thread_start_us":7201,"threads_started":1}
W20260808 03:58:41.502780 12684 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:41.484719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:57657 (local address 127.28.251.254:37713)
0808 03:58:41.492186 (+ 7467us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:41.492192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:41.492214 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:58:41.492280 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:41.492284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:41.492378 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:41.492505 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:41.493544 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.494706 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.496622 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.497353 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.497493 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:41.502419 (+ 4926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:57657: BlockingRecv error: recv got EOF from 127.28.251.193:57657 (error 108)
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":7356,"thread_start_us":6865,"threads_started":1}
Aug 08 03:58:41 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:41 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:41.953871 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:41.944320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:38731)
0808 03:58:41.945308 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:41.945331 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:41.945514 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:41.945898 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:41.945902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:41.946516 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:41.946759 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:41.946773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.948083 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.948087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:41.949072 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:41.949086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.949889 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.949895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:41.950039 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:41.950689 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:41.950709 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:41.953601 (+ 2892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":703,"threads_started":1}
W20260808 03:58:41.954041 12684 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:41.944539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:38731 (local address 127.28.251.254:37713)
0808 03:58:41.944805 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:41.944812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:41.945434 (+ 622us) server_negotiation.cc:407] Connection header received
0808 03:58:41.945646 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:41.945651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:41.945738 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:41.945862 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:41.946938 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.947928 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.949235 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:41.949707 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:41.949812 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:41.953828 (+ 4016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:38731: BlockingRecv error: recv got EOF from 127.28.251.194:38731 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 08 03:58:42 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:42 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:42 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:42 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:42.525257 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:42.503717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:49375)
0808 03:58:42.515970 (+ 12253us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:42.515999 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:42.516161 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:42.517292 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:42.517298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:42.517697 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:42.517888 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:42.517903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.519005 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.519009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:42.519997 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:42.520010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.520747 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.520753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:42.520942 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:42.521573 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:42.521602 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:42.524925 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12085,"thread_start_us":11941,"threads_started":1}
W20260808 03:58:42.525493 12690 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:42.503745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:49375 (local address 127.28.251.254:37713)
0808 03:58:42.516934 (+ 13189us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:42.516941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:42.516961 (+ 20us) server_negotiation.cc:407] Connection header received
0808 03:58:42.517016 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:42.517020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:42.517083 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:42.517182 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:42.518039 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.518852 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.520139 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.520590 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.520707 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:42.525195 (+ 4488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:49375: BlockingRecv error: recv got EOF from 127.28.251.193:49375 (error 108)
Metrics: {"server-negotiator.queue_time_us":13052,"thread_start_us":13075,"threads_started":1}
I20260808 03:58:42.525699 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:42.503667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:34467)
0808 03:58:42.516400 (+ 12733us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:42.516426 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:42.516591 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:42.517378 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:42.517383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:42.517851 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:42.518058 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:42.518072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.519283 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.519288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:42.520233 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:42.520247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.521020 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.521027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:42.521205 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:42.521672 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:42.521688 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:42.525373 (+ 3685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12561,"thread_start_us":12584,"threads_started":1}
W20260808 03:58:42.525909 12688 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:42.503749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:34467 (local address 127.28.251.254:37713)
0808 03:58:42.517080 (+ 13331us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:42.517084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:42.517101 (+ 17us) server_negotiation.cc:407] Connection header received
0808 03:58:42.517145 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:42.517149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:42.517202 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:42.517305 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:42.518211 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.519138 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.520392 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.520858 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.520967 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:42.525630 (+ 4663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:34467: BlockingRecv error: recv got EOF from 127.28.251.195:34467 (error 108)
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":13229,"thread_start_us":12864,"threads_started":1}
Aug 08 03:58:42 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:42 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:42.963649 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:42.955035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:37271)
0808 03:58:42.955679 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:42.955696 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:42.955840 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:42.956175 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:42.956180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:42.956637 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:42.956801 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:42.956811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.957826 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.957831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:42.958760 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:42.958773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.959538 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.959544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:42.959735 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:42.960360 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:42.960379 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:42.963272 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":393,"threads_started":1}
W20260808 03:58:42.963784 12688 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:42.955168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:37271 (local address 127.28.251.254:37713)
0808 03:58:42.955395 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:42.955401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:42.955789 (+ 388us) server_negotiation.cc:407] Connection header received
0808 03:58:42.955963 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:42.955967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:42.956033 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:42.956116 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:42.956960 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.957683 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.958917 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:42.959348 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:42.959491 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:42.963549 (+ 4058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:37271: BlockingRecv error: recv got EOF from 127.28.251.194:37271 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 03:58:43 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:43 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:43 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:43 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:43.549872 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:43.526672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:49679)
0808 03:58:43.540045 (+ 13373us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:43.540069 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:43.540257 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:43.540816 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:43.540822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:43.541424 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:43.541616 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:43.541626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.542797 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.542804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:43.543908 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:43.543924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.544781 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.544787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:43.544990 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:43.545727 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:43.545752 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:43.549501 (+ 3749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13216,"thread_start_us":13032,"threads_started":1}
W20260808 03:58:43.550068 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:43.526815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:49679 (local address 127.28.251.254:37713)
0808 03:58:43.539792 (+ 12977us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:43.539799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:43.540392 (+ 593us) server_negotiation.cc:407] Connection header received
0808 03:58:43.540474 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:43.540479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:43.540619 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:43.540721 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:43.541789 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.542622 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.544063 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.544617 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.544721 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:43.549753 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:49679: BlockingRecv error: recv got EOF from 127.28.251.193:49679 (error 108)
Metrics: {"server-negotiator.queue_time_us":12825,"thread_start_us":13337,"threads_started":1}
I20260808 03:58:43.550104 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:43.526935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:45657)
0808 03:58:43.540642 (+ 13707us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:43.540660 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:43.540819 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:43.541203 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:43.541208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:43.541640 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:43.541836 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:43.541851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.543129 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.543133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:43.544214 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:43.544230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.545063 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.545067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:43.545204 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:43.545912 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:43.545931 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:43.549761 (+ 3830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13598,"thread_start_us":13478,"threads_started":1}
W20260808 03:58:43.550487 12693 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:43.526972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:45657 (local address 127.28.251.254:37713)
0808 03:58:43.540494 (+ 13522us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:43.540499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:43.540762 (+ 263us) server_negotiation.cc:407] Connection header received
0808 03:58:43.540951 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:43.540954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:43.541015 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:43.541109 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:43.542020 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.542974 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.544369 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.544896 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.545014 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:43.550166 (+ 5152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:45657: BlockingRecv error: recv got EOF from 127.28.251.195:45657 (error 108)
Metrics: {"server-negotiator.queue_time_us":13327,"thread_start_us":12530,"threads_started":1}
Aug 08 03:58:43 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:43 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:43.974184 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:43.964834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:43011)
0808 03:58:43.965485 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:43.965504 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:43.965639 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:43.965962 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:43.965965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:43.966380 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:43.966552 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:43.966562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.967744 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.967748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:43.968748 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:43.968762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.969598 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.969603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:43.969743 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:43.970492 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:43.970513 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:43.973830 (+ 3317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":327,"threads_started":1}
W20260808 03:58:43.974304 12693 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:43.965121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:43011 (local address 127.28.251.254:37713)
0808 03:58:43.965347 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:43.965353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:43.965589 (+ 236us) server_negotiation.cc:407] Connection header received
0808 03:58:43.965742 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:43.965746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:43.965833 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:43.965928 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:43.966733 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.967552 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.968929 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:43.969376 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:43.969492 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:43.974065 (+ 4573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:43011: BlockingRecv error: recv got EOF from 127.28.251.194:43011 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 08 03:58:44 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:44 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:44 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:44 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:44.567124 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:44.550915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:33865)
0808 03:58:44.557517 (+ 6602us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:44.557542 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:44.557700 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:44.558719 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:44.558724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:44.559244 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:44.559511 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:44.559529 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.560716 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.560723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:44.561793 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:44.561809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.562676 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.562683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:44.562899 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:44.563658 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:44.563678 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:44.566773 (+ 3095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6450,"thread_start_us":6315,"threads_started":1}
I20260808 03:58:44.567302 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:44.550966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:35713)
0808 03:58:44.557935 (+ 6969us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:44.557956 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:44.558083 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:44.558960 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:44.558966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:44.559527 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:44.559764 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:44.559780 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.561125 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.561130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:44.562172 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:44.562188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.563062 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.563066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:44.563188 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:44.563789 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:44.563807 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:44.566983 (+ 3176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6824,"thread_start_us":6659,"threads_started":1}
W20260808 03:58:44.567337 12707 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:44.551027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:33865 (local address 127.28.251.254:37713)
0808 03:58:44.558337 (+ 7310us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:44.558344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:44.558368 (+ 24us) server_negotiation.cc:407] Connection header received
0808 03:58:44.558438 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:44.558447 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:44.558529 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:44.558638 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:44.559685 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.560554 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.561961 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.562521 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.562621 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:44.567027 (+ 4406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:33865: BlockingRecv error: recv got EOF from 127.28.251.193:33865 (error 108)
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":7140,"spinlock_wait_cycles":6016,"thread_start_us":7273,"threads_started":1}
W20260808 03:58:44.567778 12706 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:44.551027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:35713 (local address 127.28.251.254:37713)
0808 03:58:44.558559 (+ 7532us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:44.558563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:44.558578 (+ 15us) server_negotiation.cc:407] Connection header received
0808 03:58:44.558625 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:44.558630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:44.558722 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:44.558889 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:44.559913 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.560969 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.562325 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.562875 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.563003 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:44.567205 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:35713: BlockingRecv error: recv got EOF from 127.28.251.195:35713 (error 108)
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":7450,"spinlock_wait_cycles":5504,"thread_start_us":6856,"threads_started":1}
Aug 08 03:58:44 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:44 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:44.987037 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:44.975301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:35285)
0808 03:58:44.975871 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:44.975887 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:44.976032 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:44.976367 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:44.976372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:44.976792 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:44.977008 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:44.977019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.978096 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.978102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:44.979050 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:44.979064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.979843 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.979849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:44.980023 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:44.983489 (+ 3466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:44.983522 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:44.986645 (+ 3123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":312,"threads_started":1}
W20260808 03:58:44.987142 12706 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:44.975491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:35285 (local address 127.28.251.254:37713)
0808 03:58:44.975709 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:44.975716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:44.975975 (+ 259us) server_negotiation.cc:407] Connection header received
0808 03:58:44.976157 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:44.976161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:44.976226 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:44.976310 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:44.977165 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.977931 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.979204 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:44.979675 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:44.979794 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:44.986907 (+ 7113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:35285: BlockingRecv error: recv got EOF from 127.28.251.194:35285 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 03:58:45 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:45 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:45 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.195@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:45 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.193@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:45.582199 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:45.568583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.195:33663)
0808 03:58:45.571873 (+ 3290us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:45.571896 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:45.572028 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:45.572574 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:45.572580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:45.573087 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:45.573353 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:45.573369 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.574757 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.574766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:45.576281 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:45.576301 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.577246 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.577250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:45.577405 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:45.578120 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:45.578150 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:45.581797 (+ 3647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3101,"thread_start_us":2965,"threads_started":1}
I20260808 03:58:45.582379 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:45.568777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.193:39555)
0808 03:58:45.571567 (+ 2790us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:45.571596 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:45.571747 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:45.573023 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:45.573030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:45.573583 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:45.573791 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:45.573802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.575145 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.575151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:45.576622 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:45.576639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.577609 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.577617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:45.577791 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:45.578213 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:45.578239 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:45.582030 (+ 3791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2581,"thread_start_us":2435,"threads_started":1}
W20260808 03:58:45.582528 12710 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:45.568642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:33663 (local address 127.28.251.254:37713)
0808 03:58:45.572233 (+ 3591us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:45.572240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:45.572262 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:58:45.572315 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:45.572319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:45.572389 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:45.572518 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:45.573525 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.574577 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.576501 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.577076 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.577219 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:45.582020 (+ 4801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.195:33663: BlockingRecv error: recv got EOF from 127.28.251.195:33663 (error 108)
Metrics: {"server-negotiator.queue_time_us":3424,"thread_start_us":3309,"threads_started":1}
W20260808 03:58:45.583125 12713 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:45.572152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:39555 (local address 127.28.251.254:37713)
0808 03:58:45.572708 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:45.572712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:45.572727 (+ 15us) server_negotiation.cc:407] Connection header received
0808 03:58:45.572775 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:45.572778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:45.572831 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:45.572913 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:45.573976 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.574968 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.576765 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.577439 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.577524 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:45.582938 (+ 5414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.193:39555: BlockingRecv error: recv got EOF from 127.28.251.193:39555 (error 108)
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":304,"threads_started":1}
Aug 08 03:58:45 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 03:58:45 dist-test-slave-pww0 krb5kdc[10860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.251.194@KRBTEST.COM for oryx/127.28.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 03:58:45.996624 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:45.988429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37713 (local address 127.28.251.194:34883)
0808 03:58:45.989004 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:45.989021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:45.989141 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:45.989442 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:45.989445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:45.989841 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:45.990020 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:45.990029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.990995 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.990999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:45.991980 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:45.991993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.992666 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.992670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:45.992789 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:45.993489 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:45.993506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:45.996350 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.251.254:37713: Server oryx/127.28.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":309,"threads_started":1}
W20260808 03:58:45.996817 12713 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:45.988490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:34883 (local address 127.28.251.254:37713)
0808 03:58:45.988703 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:45.988710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:45.989098 (+ 388us) server_negotiation.cc:407] Connection header received
0808 03:58:45.989239 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:45.989243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:45.989306 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:45.989393 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:45.990132 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.990879 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.992099 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:45.992540 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:45.992634 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:45.996594 (+ 3960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.251.194:34883: BlockingRecv error: recv got EOF from 127.28.251.194:34883 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 03:58:46.326268 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 11912
I20260808 03:58:46.339607 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:46.327608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.28.251.193:39109)
0808 03:58:46.328207 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:46.328214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:46.328260 (+ 46us) server_negotiation.cc:407] Connection header received
0808 03:58:46.328329 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:46.328337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:46.328573 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:46.328730 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:46.330282 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:46.331494 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:46.332790 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:46.333377 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:46.333492 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:46.336107 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:46.336116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:46.336132 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:46.336172 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:46.338983 (+ 2811us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.251.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 03:58:46.339301 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40474: Request ticket server kudu/127.28.251.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":291,"threads_started":1}
W20260808 03:58:46.339725 12721 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40474: Request ticket server kudu/127.28.251.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 03:58:46.339869 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 11912: Not authorized: Client connection negotiation failed: client connection to 127.28.251.193:39109: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.251.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 03:58:46.340173 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 11912
I20260808 03:58:46.370121 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 12147
I20260808 03:58:46.382663 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:46.371240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.28.251.194:36251)
0808 03:58:46.371897 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:46.371905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:46.371941 (+ 36us) server_negotiation.cc:407] Connection header received
0808 03:58:46.372014 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:46.372020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:46.372244 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:46.372418 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:46.373740 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:46.374860 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:46.376248 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:46.376847 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:46.376950 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:46.379423 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:46.379432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:46.379447 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:46.379511 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:46.382105 (+ 2594us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.251.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 03:58:46.382414 (+ 309us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40740: Request ticket server kudu/127.28.251.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":309,"threads_started":1}
W20260808 03:58:46.382773 12722 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40740: Request ticket server kudu/127.28.251.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 03:58:46.382946 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 12147: Not authorized: Client connection negotiation failed: client connection to 127.28.251.194:36251: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.251.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 03:58:46.383114 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 12147
I20260808 03:58:46.407011 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 12383
I20260808 03:58:46.419705 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:46.408143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.28.251.195:43529)
0808 03:58:46.408793 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:46.408801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:46.408828 (+ 27us) server_negotiation.cc:407] Connection header received
0808 03:58:46.408903 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:46.408909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:46.409129 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:46.409301 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:46.410790 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:46.412081 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:46.413368 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:46.413924 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:46.414027 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:46.416573 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:46.416582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:46.416598 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:46.416644 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:46.419128 (+ 2484us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.251.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 03:58:46.419420 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32954: Request ticket server kudu/127.28.251.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":316,"threads_started":1}
W20260808 03:58:46.419811 12723 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32954: Request ticket server kudu/127.28.251.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 03:58:46.419852 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 12383: Not authorized: Client connection negotiation failed: client connection to 127.28.251.195:43529: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.251.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 03:58:46.420029 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 12383
I20260808 03:58:46.448442 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 11794
I20260808 03:58:46.452044 11898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:46.576797 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 11794
2026-08-08T03:58:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21938 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T03:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:58:46Z Disabled control of system clock
I20260808 03:58:46.680363 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:36461
--webserver_interface=127.28.251.254
--webserver_port=0
--builtin_ntp_servers=127.28.251.212:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:36461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 03:58:47.043330 12738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:47.043722 12738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:47.043836 12738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:47.051947 12738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:58:47.052088 12738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:47.052136 12738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:47.052232 12738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:58:47.052299 12738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:58:47.062211 12738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:36461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:36461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:47.063753 12738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:47.065816 12738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:47.078962 12744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:47.078915 12745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:47.078876 12747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:47.085184 12738 server_base.cc:1034] running on GCE node
I20260808 03:58:47.086166 12738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:47.087924 12738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:47.089169 12738 hybrid_clock.cc:648] HybridClock initialized: now 1786161527089119 us; error 48 us; skew 500 ppm
I20260808 03:58:47.089557 12738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:58:47.092192 12738 webserver.cc:460] Webserver started at http://127.28.251.254:38517/ using document root <none> and password file <none>
I20260808 03:58:47.093003 12738 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:47.093122 12738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:47.093466 12738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:47.096700 12738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "62109a1f4f854df7b33a99788cf35b49"
format_stamp: "Formatted at 2026-08-08 03:58:47 on dist-test-slave-pww0"
server_key: "2c59c7df1f5d82733b33156fe9409a01"
server_key_iv: "d5dbaa6a28bd7bef77cba68938eb20f4"
server_key_version: "encryptionkey@0"
I20260808 03:58:47.097575 12738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "62109a1f4f854df7b33a99788cf35b49"
format_stamp: "Formatted at 2026-08-08 03:58:47 on dist-test-slave-pww0"
server_key: "2c59c7df1f5d82733b33156fe9409a01"
server_key_iv: "d5dbaa6a28bd7bef77cba68938eb20f4"
server_key_version: "encryptionkey@0"
I20260808 03:58:47.103256 12738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 03:58:47.107079 12753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:47.108734 12738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:47.108948 12738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "62109a1f4f854df7b33a99788cf35b49"
format_stamp: "Formatted at 2026-08-08 03:58:47 on dist-test-slave-pww0"
server_key: "2c59c7df1f5d82733b33156fe9409a01"
server_key_iv: "d5dbaa6a28bd7bef77cba68938eb20f4"
server_key_version: "encryptionkey@0"
I20260808 03:58:47.109177 12738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:47.125453 12738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:47.126622 12738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:47.126982 12738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:47.147975 12738 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:36461
I20260808 03:58:47.148000 12844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:36461 every 8 connection(s)
I20260808 03:58:47.149436 12738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:58:47.153311 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 12738
I20260808 03:58:47.153523 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:58:47.153855 12845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:47.154407 29679 external_mini_cluster.cc:1428] Setting key 0673edf53577a85911193f45c36ab02b
I20260808 03:58:47.168529 12845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49: Bootstrap starting.
I20260808 03:58:47.170256 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:47.156730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.28.251.254:36461)
0808 03:58:47.157836 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:47.157849 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:47.157899 (+ 50us) server_negotiation.cc:407] Connection header received
0808 03:58:47.160384 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:47.160410 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:47.160970 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:47.161341 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:47.162100 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:47.163429 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:47.165424 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:47.166262 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:47.166411 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:47.167784 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:47.167792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:47.167810 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:47.168395 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:47.168478 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:47.168502 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:47.168594 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:47.168875 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 03:58:47.169228 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":385,"threads_started":1}
I20260808 03:58:47.172415 12845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:47.173868 12845 log.cc:824] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:47.177276 12845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49: No bootstrap required, opened a new log
I20260808 03:58:47.185341 12845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62109a1f4f854df7b33a99788cf35b49" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36461 } }
I20260808 03:58:47.185781 12845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:47.185865 12845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62109a1f4f854df7b33a99788cf35b49, State: Initialized, Role: FOLLOWER
I20260808 03:58:47.186463 12845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62109a1f4f854df7b33a99788cf35b49" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36461 } }
I20260808 03:58:47.186690 12845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:58:47.186797 12845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:58:47.186933 12845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:47.189220 12845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62109a1f4f854df7b33a99788cf35b49" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36461 } }
I20260808 03:58:47.189720 12845 leader_election.cc:302] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62109a1f4f854df7b33a99788cf35b49; no voters:
I20260808 03:58:47.190325 12845 leader_election.cc:288] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:58:47.190501 12853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:47.191226 12853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [term 1 LEADER]: Becoming Leader. State: Replica: 62109a1f4f854df7b33a99788cf35b49, State: Running, Role: LEADER
I20260808 03:58:47.191838 12853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62109a1f4f854df7b33a99788cf35b49" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36461 } }
I20260808 03:58:47.192868 12845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:58:47.194976 12855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62109a1f4f854df7b33a99788cf35b49. Latest consensus state: current_term: 1 leader_uuid: "62109a1f4f854df7b33a99788cf35b49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62109a1f4f854df7b33a99788cf35b49" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36461 } } }
I20260808 03:58:47.195516 12855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:47.194985 12854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62109a1f4f854df7b33a99788cf35b49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62109a1f4f854df7b33a99788cf35b49" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36461 } } }
I20260808 03:58:47.196386 12854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:47.199196 12862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:58:47.204352 12862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:58:47.214340 12862 catalog_manager.cc:1347] Generated new cluster ID: ffe1f7a58308466bad13b0d9712ead57
I20260808 03:58:47.214496 12862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:58:47.231206 12862 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:58:47.232729 12862 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:58:47.242851 12862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62109a1f4f854df7b33a99788cf35b49: Generated new TSK 0
I20260808 03:58:47.243842 12862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 03:58:47.260982 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36461
--builtin_ntp_servers=127.28.251.212:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 03:58:47.631770 12881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:47.632087 12881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:47.632220 12881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:47.639895 12881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:47.640021 12881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:47.640169 12881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:58:47.649974 12881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:47.651379 12881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:47.653486 12881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:47.667762 12887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:47.667832 12888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:47.668016 12891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:47.669090 12881 server_base.cc:1034] running on GCE node
I20260808 03:58:47.669866 12881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:47.671046 12881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:47.672287 12881 hybrid_clock.cc:648] HybridClock initialized: now 1786161527672222 us; error 53 us; skew 500 ppm
I20260808 03:58:47.672660 12881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:58:47.675504 12881 webserver.cc:460] Webserver started at http://127.28.251.193:36967/ using document root <none> and password file <none>
I20260808 03:58:47.676322 12881 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:47.676443 12881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:47.676808 12881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:47.680145 12881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "d802287185dd48ab972d0c67e046f535"
format_stamp: "Formatted at 2026-08-08 03:58:47 on dist-test-slave-pww0"
server_key: "6e19f1fd521514597f82377fb4db0498"
server_key_iv: "ff634f00e5e7905552a1e93e00a13834"
server_key_version: "encryptionkey@0"
I20260808 03:58:47.681090 12881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "d802287185dd48ab972d0c67e046f535"
format_stamp: "Formatted at 2026-08-08 03:58:47 on dist-test-slave-pww0"
server_key: "6e19f1fd521514597f82377fb4db0498"
server_key_iv: "ff634f00e5e7905552a1e93e00a13834"
server_key_version: "encryptionkey@0"
I20260808 03:58:47.687088 12881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260808 03:58:47.691162 12901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:47.692737 12881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:47.692948 12881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "d802287185dd48ab972d0c67e046f535"
format_stamp: "Formatted at 2026-08-08 03:58:47 on dist-test-slave-pww0"
server_key: "6e19f1fd521514597f82377fb4db0498"
server_key_iv: "ff634f00e5e7905552a1e93e00a13834"
server_key_version: "encryptionkey@0"
I20260808 03:58:47.693202 12881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:47.718021 12881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:47.719041 12881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:47.719380 12881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:47.720571 12881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:47.722263 12881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:47.722373 12881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:47.722486 12881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:47.722566 12881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:47.755785 12881 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:36113
I20260808 03:58:47.755844 13078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:36113 every 8 connection(s)
I20260808 03:58:47.757192 12881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:58:47.765106 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 12881
I20260808 03:58:47.765324 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:58:47.766003 29679 external_mini_cluster.cc:1428] Setting key 4433dbd7783f3e7355a81d559ef12eb2
I20260808 03:58:47.769423 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36461
--builtin_ntp_servers=127.28.251.212:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 03:58:47.770901 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:47.759854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:43499 (local address 127.28.251.254:36461)
0808 03:58:47.760299 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:47.760304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:47.761574 (+ 1270us) server_negotiation.cc:407] Connection header received
0808 03:58:47.763450 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:47.763487 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:47.763576 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:47.763679 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:47.764890 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:47.765669 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:47.767555 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:47.768048 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:47.768185 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:47.769998 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:47.770004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:47.770007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:47.770028 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:47.770074 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:47.770088 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:47.770168 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:47.770504 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 03:58:47.770606 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
I20260808 03:58:47.772411 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:47.760425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:36461 (local address 127.28.251.193:43499)
0808 03:58:47.761399 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:47.761445 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:47.763140 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:47.763935 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:47.763944 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:47.764046 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:47.764697 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:47.764716 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:47.765842 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:47.765847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:47.767358 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:47.767376 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:47.768322 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:47.768332 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:47.768671 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:47.769546 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:58:47.769551 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:47.769619 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:58:47.769748 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:47.770193 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:47.770205 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:47.770213 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 03:58:47.770428 (+ 215us) client_negotiation.cc:240] Negotiation successful
0808 03:58:47.770914 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":304,"threads_started":1}
I20260808 03:58:47.774098 13081 heartbeater.cc:347] Connected to a master server at 127.28.251.254:36461
I20260808 03:58:47.774421 13081 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:47.775243 13081 heartbeater.cc:511] Master 127.28.251.254:36461 requested a full tablet report, sending...
I20260808 03:58:47.777750 12789 ts_manager.cc:194] Registered new tserver with Master: d802287185dd48ab972d0c67e046f535 (127.28.251.193:36113)
I20260808 03:58:47.779784 12789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.193:43499
W20260808 03:58:48.134262 13087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:48.134570 13087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:48.134704 13087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:48.142205 13087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:48.142325 13087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:48.142485 13087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:58:48.152316 13087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:48.153723 13087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:48.155691 13087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:48.169857 13096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:48.169924 13098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:48.169885 13102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:48.170563 13087 server_base.cc:1034] running on GCE node
I20260808 03:58:48.171265 13087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:48.172349 13087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:48.173576 13087 hybrid_clock.cc:648] HybridClock initialized: now 1786161528173513 us; error 56 us; skew 500 ppm
I20260808 03:58:48.173920 13087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:58:48.176456 13087 webserver.cc:460] Webserver started at http://127.28.251.194:42209/ using document root <none> and password file <none>
I20260808 03:58:48.177217 13087 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:48.177342 13087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:48.177690 13087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:48.180845 13087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "bca31f5270af43ac89d5b8da70d48c60"
format_stamp: "Formatted at 2026-08-08 03:58:48 on dist-test-slave-pww0"
server_key: "4da2389fa6c3cfc6ccba1d0637ca5673"
server_key_iv: "bee260da0dc0669ba69b2acbdbd2a06e"
server_key_version: "encryptionkey@0"
I20260808 03:58:48.181696 13087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "bca31f5270af43ac89d5b8da70d48c60"
format_stamp: "Formatted at 2026-08-08 03:58:48 on dist-test-slave-pww0"
server_key: "4da2389fa6c3cfc6ccba1d0637ca5673"
server_key_iv: "bee260da0dc0669ba69b2acbdbd2a06e"
server_key_version: "encryptionkey@0"
I20260808 03:58:48.187305 13087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 03:58:48.191273 13111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:48.192840 13087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:48.193053 13087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "bca31f5270af43ac89d5b8da70d48c60"
format_stamp: "Formatted at 2026-08-08 03:58:48 on dist-test-slave-pww0"
server_key: "4da2389fa6c3cfc6ccba1d0637ca5673"
server_key_iv: "bee260da0dc0669ba69b2acbdbd2a06e"
server_key_version: "encryptionkey@0"
I20260808 03:58:48.193312 13087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:48.213328 13087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:48.214332 13087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:48.214653 13087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:48.215751 13087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:48.217268 13087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:48.217375 13087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:48.217482 13087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:48.217553 13087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:48.248878 13087 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:35307
I20260808 03:58:48.248937 13291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:35307 every 8 connection(s)
I20260808 03:58:48.250294 13087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:58:48.253587 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 13087
I20260808 03:58:48.253750 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:58:48.254280 29679 external_mini_cluster.cc:1428] Setting key 678812b58ce9e5ece690372c1de07c59
I20260808 03:58:48.257719 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36461
--builtin_ntp_servers=127.28.251.212:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 03:58:48.264920 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:48.253090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:34209 (local address 127.28.251.254:36461)
0808 03:58:48.253322 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:48.253328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:48.255049 (+ 1721us) server_negotiation.cc:407] Connection header received
0808 03:58:48.257279 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:48.257284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:48.257354 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:48.257443 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:48.258965 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.259805 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.261955 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.262425 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.262559 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:48.264060 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:48.264066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:48.264069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:48.264093 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:48.264128 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:48.264141 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:48.264218 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:48.264452 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 03:58:48.264563 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 03:58:48.266325 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:48.253677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:36461 (local address 127.28.251.194:34209)
0808 03:58:48.254851 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:48.254904 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:48.256941 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:48.257739 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:48.257752 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:48.257882 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:48.258728 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:48.258755 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.259972 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.259977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:48.261775 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:48.261799 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.262709 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.262718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:48.262960 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:48.263723 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:58:48.263726 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:48.263770 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:58:48.263858 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:48.264235 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:48.264248 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:48.264256 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 03:58:48.264427 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 03:58:48.264879 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":313,"threads_started":1}
I20260808 03:58:48.267957 13292 heartbeater.cc:347] Connected to a master server at 127.28.251.254:36461
I20260808 03:58:48.268328 13292 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:48.269349 13292 heartbeater.cc:511] Master 127.28.251.254:36461 requested a full tablet report, sending...
I20260808 03:58:48.271018 12789 ts_manager.cc:194] Registered new tserver with Master: bca31f5270af43ac89d5b8da70d48c60 (127.28.251.194:35307)
I20260808 03:58:48.271893 12789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.194:34209
W20260808 03:58:48.618410 13298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:48.618741 13298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:48.618855 13298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:48.626335 13298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:48.626463 13298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:48.626606 13298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:58:48.636166 13298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:48.637593 13298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:48.639446 13298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:48.653542 13320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:48.653512 13312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:48.653602 13315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:48.654498 13298 server_base.cc:1034] running on GCE node
I20260808 03:58:48.655166 13298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:48.656246 13298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:48.657498 13298 hybrid_clock.cc:648] HybridClock initialized: now 1786161528657435 us; error 49 us; skew 500 ppm
I20260808 03:58:48.657851 13298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:58:48.660343 13298 webserver.cc:460] Webserver started at http://127.28.251.195:39927/ using document root <none> and password file <none>
I20260808 03:58:48.661151 13298 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:48.661262 13298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:48.661610 13298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:48.664747 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "14fbe47750f14e0f8b9e7097868b744e"
format_stamp: "Formatted at 2026-08-08 03:58:48 on dist-test-slave-pww0"
server_key: "78a39b9d67dd71f28fa701e42f762360"
server_key_iv: "8dae9d46b8d60f6b0bffa4b63a2f1a28"
server_key_version: "encryptionkey@0"
I20260808 03:58:48.665611 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "14fbe47750f14e0f8b9e7097868b744e"
format_stamp: "Formatted at 2026-08-08 03:58:48 on dist-test-slave-pww0"
server_key: "78a39b9d67dd71f28fa701e42f762360"
server_key_iv: "8dae9d46b8d60f6b0bffa4b63a2f1a28"
server_key_version: "encryptionkey@0"
I20260808 03:58:48.671108 13298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 03:58:48.674782 13327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:48.676205 13298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:48.676381 13298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "14fbe47750f14e0f8b9e7097868b744e"
format_stamp: "Formatted at 2026-08-08 03:58:48 on dist-test-slave-pww0"
server_key: "78a39b9d67dd71f28fa701e42f762360"
server_key_iv: "8dae9d46b8d60f6b0bffa4b63a2f1a28"
server_key_version: "encryptionkey@0"
I20260808 03:58:48.676607 13298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:48.714891 13298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:48.715879 13298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:48.716193 13298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:48.717242 13298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:48.718806 13298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:48.718919 13298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:48.719033 13298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:48.719105 13298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:48.751262 13298 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:35729
I20260808 03:58:48.751299 13517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:35729 every 8 connection(s)
I20260808 03:58:48.752660 13298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:58:48.761790 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 13298
I20260808 03:58:48.761981 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:58:48.762562 29679 external_mini_cluster.cc:1428] Setting key 5289b1b74df75bd8a58d2bce055c094a
I20260808 03:58:48.766382 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:48.755361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:38109 (local address 127.28.251.254:36461)
0808 03:58:48.755603 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:48.755610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:48.757227 (+ 1617us) server_negotiation.cc:407] Connection header received
0808 03:58:48.758998 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:48.759004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:48.759097 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:48.759226 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:48.760348 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.761104 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.763375 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.763901 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.764026 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:48.765596 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:48.765602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:48.765607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:48.765629 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:48.765671 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:48.765687 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:48.765779 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:48.766033 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 03:58:48.766121 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 03:58:48.767442 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:48.755944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:36461 (local address 127.28.251.195:38109)
0808 03:58:48.757028 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:48.757082 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:48.758703 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:48.759439 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:48.759447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:48.759577 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:48.760172 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:48.760191 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.761245 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.761249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:48.763198 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:48.763213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.764156 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.764164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:48.764489 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:48.765260 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:58:48.765263 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:48.765306 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:58:48.765404 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:48.765781 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:48.765792 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:48.765798 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 03:58:48.765995 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 03:58:48.766369 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":330,"threads_started":1}
I20260808 03:58:48.768890 13518 heartbeater.cc:347] Connected to a master server at 127.28.251.254:36461
I20260808 03:58:48.769244 13518 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:48.770056 13518 heartbeater.cc:511] Master 127.28.251.254:36461 requested a full tablet report, sending...
I20260808 03:58:48.771513 12789 ts_manager.cc:194] Registered new tserver with Master: 14fbe47750f14e0f8b9e7097868b744e (127.28.251.195:35729)
I20260808 03:58:48.772307 12789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.195:38109
I20260808 03:58:48.778447 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 03:58:48.784090 13081 heartbeater.cc:503] Master 127.28.251.254:36461 was elected leader, sending a full tablet report...
W20260808 03:58:48.785056 13083 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:48.783251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.28.251.254:36461)
0808 03:58:48.783522 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:48.783528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:48.783999 (+ 471us) server_negotiation.cc:407] Connection header received
0808 03:58:48.784145 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:48.784151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:48.784239 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:48.784353 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:48.784783 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41040: BlockingRecv error: recv got EOF from 127.0.0.1:41040 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 03:58:48.785959 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 12881
I20260808 03:58:48.793840 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:48.787018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.28.251.193:36113)
0808 03:58:48.787601 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:48.787609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:48.787636 (+ 27us) server_negotiation.cc:407] Connection header received
0808 03:58:48.787688 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:48.787694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:48.787907 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:48.788048 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:48.788400 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.789419 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.790690 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.791284 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.791379 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:48.792397 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:48.792405 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:48.792420 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:48.793047 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:48.793109 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:48.793130 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:48.793227 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:48.793342 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 03:58:48.793467 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":263,"threads_started":1}
I20260808 03:58:48.795109 13074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:48.879904 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 12881
I20260808 03:58:48.908654 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 13087
I20260808 03:58:48.917025 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:48.909720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.28.251.194:35307)
0808 03:58:48.910321 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:48.910328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:48.910371 (+ 43us) server_negotiation.cc:407] Connection header received
0808 03:58:48.910437 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:48.910443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:48.910666 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:48.910804 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:48.911240 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.912328 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.913638 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:48.914343 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:48.914450 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:48.915552 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:48.915561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:48.915576 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:48.916216 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:48.916269 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:48.916292 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:48.916386 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:48.916547 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 03:58:48.916685 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":288,"threads_started":1}
I20260808 03:58:48.918431 13286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:49.002228 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 13087
I20260808 03:58:49.029458 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 13298
I20260808 03:58:49.038475 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:49.030529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.28.251.195:35729)
0808 03:58:49.031261 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:49.031268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:49.031300 (+ 32us) server_negotiation.cc:407] Connection header received
0808 03:58:49.031367 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:49.031374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:49.031675 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:49.031811 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:49.032322 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:49.033409 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:49.034827 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:49.035446 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:49.035640 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:49.036786 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:49.036796 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:49.036811 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:49.037451 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:49.037503 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:49.037525 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:49.037618 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:49.037900 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 03:58:49.038073 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":391,"threads_started":1}
I20260808 03:58:49.039970 13511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:49.124488 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 13298
I20260808 03:58:49.153851 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 12738
I20260808 03:58:49.155216 12835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:49.261890 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 12738
2026-08-08T03:58:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2681 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T03:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:58:49Z Disabled control of system clock
I20260808 03:58:49.361043 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40305
--webserver_interface=127.28.251.254
--webserver_port=0
--builtin_ntp_servers=127.28.251.212:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:40305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 03:58:49.717435 13551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:49.717759 13551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:49.717844 13551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:49.725435 13551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:58:49.725562 13551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:49.725600 13551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:49.725658 13551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:58:49.725696 13551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:58:49.735263 13551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:40305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:49.736753 13551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:49.738739 13551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:49.751042 13556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:49.751086 13558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:49.751117 13563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:49.752163 13551 server_base.cc:1034] running on GCE node
I20260808 03:58:49.753034 13551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:49.754572 13551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:49.755780 13551 hybrid_clock.cc:648] HybridClock initialized: now 1786161529755738 us; error 30 us; skew 500 ppm
I20260808 03:58:49.756139 13551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:58:49.758541 13551 webserver.cc:460] Webserver started at http://127.28.251.254:44587/ using document root <none> and password file <none>
I20260808 03:58:49.759280 13551 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:49.759404 13551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:49.759765 13551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:49.765105 13551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "27451d9b976d4f47b3cab60a71de5d32"
format_stamp: "Formatted at 2026-08-08 03:58:49 on dist-test-slave-pww0"
server_key: "1b44e640312c107b112100b90af96946"
server_key_iv: "cff9c240b6c9b1abaac09510d2e85f40"
server_key_version: "encryptionkey@0"
I20260808 03:58:49.766024 13551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "27451d9b976d4f47b3cab60a71de5d32"
format_stamp: "Formatted at 2026-08-08 03:58:49 on dist-test-slave-pww0"
server_key: "1b44e640312c107b112100b90af96946"
server_key_iv: "cff9c240b6c9b1abaac09510d2e85f40"
server_key_version: "encryptionkey@0"
I20260808 03:58:49.771878 13551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260808 03:58:49.775641 13573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:49.777135 13551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:58:49.777320 13551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "27451d9b976d4f47b3cab60a71de5d32"
format_stamp: "Formatted at 2026-08-08 03:58:49 on dist-test-slave-pww0"
server_key: "1b44e640312c107b112100b90af96946"
server_key_iv: "cff9c240b6c9b1abaac09510d2e85f40"
server_key_version: "encryptionkey@0"
I20260808 03:58:49.777554 13551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:49.783557 13551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:49.784373 13551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:49.784664 13551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:49.802951 13551 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:40305
I20260808 03:58:49.802970 13656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:40305 every 8 connection(s)
I20260808 03:58:49.804546 13551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:58:49.808701 13659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:49.813786 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 13551
I20260808 03:58:49.814014 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:58:49.814658 29679 external_mini_cluster.cc:1428] Setting key 316ecc6a1b063a513b0b2a9320d3436c
I20260808 03:58:49.820367 13659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32: Bootstrap starting.
I20260808 03:58:49.823390 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:49.816774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.28.251.254:40305)
0808 03:58:49.817877 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:49.817889 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:49.817935 (+ 46us) server_negotiation.cc:407] Connection header received
0808 03:58:49.819665 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:49.819690 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:49.820140 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:49.820542 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:49.821718 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:49.821725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:49.821739 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:49.821754 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:49.821815 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:49.821830 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:49.822011 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:49.822120 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 03:58:49.822412 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":377,"threads_started":1}
I20260808 03:58:49.824355 13659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:49.825608 13659 log.cc:824] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:49.828827 13659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32: No bootstrap required, opened a new log
I20260808 03:58:49.836642 13659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27451d9b976d4f47b3cab60a71de5d32" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40305 } }
I20260808 03:58:49.837069 13659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:49.837188 13659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27451d9b976d4f47b3cab60a71de5d32, State: Initialized, Role: FOLLOWER
I20260808 03:58:49.837811 13659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27451d9b976d4f47b3cab60a71de5d32" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40305 } }
I20260808 03:58:49.838052 13659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:58:49.838225 13659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:58:49.838424 13659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:49.840855 13659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27451d9b976d4f47b3cab60a71de5d32" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40305 } }
I20260808 03:58:49.841409 13659 leader_election.cc:302] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27451d9b976d4f47b3cab60a71de5d32; no voters:
I20260808 03:58:49.842016 13659 leader_election.cc:288] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:58:49.842180 13664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:49.842901 13664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [term 1 LEADER]: Becoming Leader. State: Replica: 27451d9b976d4f47b3cab60a71de5d32, State: Running, Role: LEADER
I20260808 03:58:49.843513 13664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27451d9b976d4f47b3cab60a71de5d32" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40305 } }
I20260808 03:58:49.844192 13659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:58:49.846176 13666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27451d9b976d4f47b3cab60a71de5d32. Latest consensus state: current_term: 1 leader_uuid: "27451d9b976d4f47b3cab60a71de5d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27451d9b976d4f47b3cab60a71de5d32" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40305 } } }
I20260808 03:58:49.846581 13666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:49.848284 13669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:58:49.848358 13665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27451d9b976d4f47b3cab60a71de5d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27451d9b976d4f47b3cab60a71de5d32" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40305 } } }
I20260808 03:58:49.849581 13665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32 [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:49.854220 13669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:58:49.864755 13669 catalog_manager.cc:1347] Generated new cluster ID: e924a03038b04d78919195f9f5e0d2bc
I20260808 03:58:49.864897 13669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:58:49.890440 13669 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:58:49.891027 13669 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:58:49.899943 13669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27451d9b976d4f47b3cab60a71de5d32: Generated new TSK 0
I20260808 03:58:49.900831 13669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 03:58:49.915693 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40305
--builtin_ntp_servers=127.28.251.212:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 03:58:50.314456 13694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:50.314769 13694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:50.314873 13694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:50.322337 13694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:50.322468 13694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:50.322616 13694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:58:50.332356 13694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:50.333781 13694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:50.335870 13694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:50.350042 13702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:50.350145 13703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:50.350590 13705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:50.350881 13694 server_base.cc:1034] running on GCE node
I20260808 03:58:50.351541 13694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:50.352756 13694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:50.353981 13694 hybrid_clock.cc:648] HybridClock initialized: now 1786161530353929 us; error 51 us; skew 500 ppm
I20260808 03:58:50.354328 13694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:58:50.360304 13694 webserver.cc:460] Webserver started at http://127.28.251.193:45549/ using document root <none> and password file <none>
I20260808 03:58:50.361119 13694 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:50.361239 13694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:50.361609 13694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:50.364861 13694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "ec38bee79f0e46e3846c90571f5485f1"
format_stamp: "Formatted at 2026-08-08 03:58:50 on dist-test-slave-pww0"
server_key: "6593d00b9c9b056a9fd146c750194e9b"
server_key_iv: "71d758abbdce8e95b3916a70e234de08"
server_key_version: "encryptionkey@0"
I20260808 03:58:50.365693 13694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "ec38bee79f0e46e3846c90571f5485f1"
format_stamp: "Formatted at 2026-08-08 03:58:50 on dist-test-slave-pww0"
server_key: "6593d00b9c9b056a9fd146c750194e9b"
server_key_iv: "71d758abbdce8e95b3916a70e234de08"
server_key_version: "encryptionkey@0"
I20260808 03:58:50.371351 13694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 03:58:50.375200 13717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:50.376750 13694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 03:58:50.376956 13694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "ec38bee79f0e46e3846c90571f5485f1"
format_stamp: "Formatted at 2026-08-08 03:58:50 on dist-test-slave-pww0"
server_key: "6593d00b9c9b056a9fd146c750194e9b"
server_key_iv: "71d758abbdce8e95b3916a70e234de08"
server_key_version: "encryptionkey@0"
I20260808 03:58:50.377190 13694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:50.383598 13694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:50.384508 13694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:50.384785 13694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:50.385797 13694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:50.387347 13694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:50.387441 13694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:50.387590 13694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:50.387665 13694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:50.419924 13694 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:36673
I20260808 03:58:50.419973 13894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:36673 every 8 connection(s)
I20260808 03:58:50.421329 13694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:58:50.425200 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 13694
I20260808 03:58:50.425444 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:58:50.426203 29679 external_mini_cluster.cc:1428] Setting key 4fb9fa21b6b12f40b5fb6ced7a3364b1
I20260808 03:58:50.429075 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40305
--builtin_ntp_servers=127.28.251.212:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 03:58:50.430068 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:50.424011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:44603 (local address 127.28.251.254:40305)
0808 03:58:50.424490 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:50.424495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:50.425794 (+ 1299us) server_negotiation.cc:407] Connection header received
0808 03:58:50.427757 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:50.427763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:50.427830 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:50.427929 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:50.429298 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:50.429304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:50.429308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:50.429336 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:50.429390 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:50.429406 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:50.429529 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:50.429695 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 03:58:50.429760 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":228,"threads_started":1}
I20260808 03:58:50.430974 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:50.424543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40305 (local address 127.28.251.193:44603)
0808 03:58:50.425624 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:50.425672 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:50.427448 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:50.428188 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:50.428197 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:50.428294 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:50.428956 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:58:50.428959 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:50.429000 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:58:50.429096 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:50.429513 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:50.429521 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:50.429527 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 03:58:50.429747 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 03:58:50.429972 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":340,"threads_started":1}
I20260808 03:58:50.432595 13899 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40305
I20260808 03:58:50.433044 13899 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:50.433712 13899 heartbeater.cc:511] Master 127.28.251.254:40305 requested a full tablet report, sending...
I20260808 03:58:50.435866 13604 ts_manager.cc:194] Registered new tserver with Master: ec38bee79f0e46e3846c90571f5485f1 (127.28.251.193:36673)
W20260808 03:58:50.790496 13904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:50.790799 13904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:50.790940 13904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:50.798405 13904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:50.798528 13904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:50.798689 13904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:58:50.808209 13904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:50.809540 13904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:50.811507 13904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:50.825124 13914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:50.825090 13918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:50.825192 13915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:50.825897 13904 server_base.cc:1034] running on GCE node
I20260808 03:58:50.826598 13904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:50.827755 13904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:50.828996 13904 hybrid_clock.cc:648] HybridClock initialized: now 1786161530828940 us; error 51 us; skew 500 ppm
I20260808 03:58:50.829344 13904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:58:50.831887 13904 webserver.cc:460] Webserver started at http://127.28.251.194:45355/ using document root <none> and password file <none>
I20260808 03:58:50.832661 13904 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:50.832774 13904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:50.833128 13904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:50.836341 13904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "e99440b1de76433a8a8b0b27424ab219"
format_stamp: "Formatted at 2026-08-08 03:58:50 on dist-test-slave-pww0"
server_key: "7a62d367be31800169dbdf7ad96614f6"
server_key_iv: "8474a8884dc3e057acbf3c8bf05ee838"
server_key_version: "encryptionkey@0"
I20260808 03:58:50.837215 13904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "e99440b1de76433a8a8b0b27424ab219"
format_stamp: "Formatted at 2026-08-08 03:58:50 on dist-test-slave-pww0"
server_key: "7a62d367be31800169dbdf7ad96614f6"
server_key_iv: "8474a8884dc3e057acbf3c8bf05ee838"
server_key_version: "encryptionkey@0"
I20260808 03:58:50.842938 13904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 03:58:50.846969 13925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:50.848572 13904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:50.848755 13904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "e99440b1de76433a8a8b0b27424ab219"
format_stamp: "Formatted at 2026-08-08 03:58:50 on dist-test-slave-pww0"
server_key: "7a62d367be31800169dbdf7ad96614f6"
server_key_iv: "8474a8884dc3e057acbf3c8bf05ee838"
server_key_version: "encryptionkey@0"
I20260808 03:58:50.849007 13904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:50.855513 13904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:50.856436 13904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:50.856717 13904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:50.857796 13904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:50.859340 13904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:50.859441 13904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:50.859582 13904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:50.859674 13904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:50.891994 13904 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:42363
I20260808 03:58:50.892052 14107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:42363 every 8 connection(s)
I20260808 03:58:50.893447 13904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:58:50.902194 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:50.896336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:47177 (local address 127.28.251.254:40305)
0808 03:58:50.896565 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:50.896570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:50.898042 (+ 1472us) server_negotiation.cc:407] Connection header received
0808 03:58:50.899896 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:50.899902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:50.899976 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:50.900068 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:50.901437 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:50.901443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:50.901448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:50.901471 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:50.901505 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:50.901517 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:50.901650 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:50.901838 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 03:58:50.901914 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 03:58:50.902760 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 13904
I20260808 03:58:50.902952 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:58:50.903312 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:50.896840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40305 (local address 127.28.251.194:47177)
0808 03:58:50.897872 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:50.897913 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:50.899598 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:50.900350 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:50.900359 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:50.900461 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:50.901111 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:58:50.901114 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:50.901156 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:58:50.901251 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:50.901613 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:50.901626 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:50.901634 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 03:58:50.901924 (+ 290us) client_negotiation.cc:240] Negotiation successful
0808 03:58:50.902213 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":314,"threads_started":1}
I20260808 03:58:50.903388 29679 external_mini_cluster.cc:1428] Setting key 5048f94d941baa2b43f1f550f34c3edc
I20260808 03:58:50.905025 14108 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40305
I20260808 03:58:50.905356 14108 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:50.906008 14108 heartbeater.cc:511] Master 127.28.251.254:40305 requested a full tablet report, sending...
I20260808 03:58:50.906394 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40305
--builtin_ntp_servers=127.28.251.212:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 03:58:50.907579 13604 ts_manager.cc:194] Registered new tserver with Master: e99440b1de76433a8a8b0b27424ab219 (127.28.251.194:42363)
W20260808 03:58:51.267412 14111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:51.267820 14111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:51.268092 14111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:51.275872 14111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:51.275996 14111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:51.276146 14111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:58:51.285776 14111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:51.287205 14111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:51.289151 14111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:51.303016 14119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:51.302978 14121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:51.303051 14123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:51.304003 14111 server_base.cc:1034] running on GCE node
I20260808 03:58:51.304714 14111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:51.305797 14111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:51.307035 14111 hybrid_clock.cc:648] HybridClock initialized: now 1786161531306987 us; error 40 us; skew 500 ppm
I20260808 03:58:51.307381 14111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 03:58:51.309903 14111 webserver.cc:460] Webserver started at http://127.28.251.195:43605/ using document root <none> and password file <none>
I20260808 03:58:51.310695 14111 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:51.310813 14111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:51.311163 14111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:51.314352 14111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "ab608e4261bb447cbdabf47759ce35d2"
format_stamp: "Formatted at 2026-08-08 03:58:51 on dist-test-slave-pww0"
server_key: "5ace9ca1fe3fc7f3a0b1ec3ddd2189c8"
server_key_iv: "ab06b810959914ba514bbc73db02630a"
server_key_version: "encryptionkey@0"
I20260808 03:58:51.315228 14111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "ab608e4261bb447cbdabf47759ce35d2"
format_stamp: "Formatted at 2026-08-08 03:58:51 on dist-test-slave-pww0"
server_key: "5ace9ca1fe3fc7f3a0b1ec3ddd2189c8"
server_key_iv: "ab06b810959914ba514bbc73db02630a"
server_key_version: "encryptionkey@0"
I20260808 03:58:51.320834 14111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 03:58:51.324702 14134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:51.326181 14111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:58:51.326354 14111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "ab608e4261bb447cbdabf47759ce35d2"
format_stamp: "Formatted at 2026-08-08 03:58:51 on dist-test-slave-pww0"
server_key: "5ace9ca1fe3fc7f3a0b1ec3ddd2189c8"
server_key_iv: "ab06b810959914ba514bbc73db02630a"
server_key_version: "encryptionkey@0"
I20260808 03:58:51.326591 14111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:51.332719 14111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:51.333588 14111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:51.333873 14111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:51.334892 14111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:51.336418 14111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:51.336519 14111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:51.336628 14111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:51.336699 14111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:51.369216 14111 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:43227
I20260808 03:58:51.369266 14319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:43227 every 8 connection(s)
I20260808 03:58:51.370713 14111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:58:51.379030 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 14111
I20260808 03:58:51.379244 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:58:51.379354 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:51.373550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:51901 (local address 127.28.251.254:40305)
0808 03:58:51.373812 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:51.373816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:51.375310 (+ 1494us) server_negotiation.cc:407] Connection header received
0808 03:58:51.377105 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:51.377109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:51.377170 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:51.377263 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:51.378616 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:51.378621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:51.378625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:51.378644 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:51.378677 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:51.378692 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:51.378820 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:51.378970 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 03:58:51.379040 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260808 03:58:51.379817 29679 external_mini_cluster.cc:1428] Setting key 70e4b68bd415edd98a9bc617f70ba3e2
I20260808 03:58:51.380375 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:51.374104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40305 (local address 127.28.251.195:51901)
0808 03:58:51.375134 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:51.375178 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:51.376843 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:51.377471 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:51.377480 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:51.377581 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:51.378314 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 03:58:51.378317 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:51.378358 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 03:58:51.378455 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:51.378788 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:51.378795 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:51.378800 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 03:58:51.379022 (+ 222us) client_negotiation.cc:240] Negotiation successful
0808 03:58:51.379278 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":298,"threads_started":1}
I20260808 03:58:51.381765 14321 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40305
I20260808 03:58:51.382141 14321 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:51.382915 14321 heartbeater.cc:511] Master 127.28.251.254:40305 requested a full tablet report, sending...
I20260808 03:58:51.384359 13604 ts_manager.cc:194] Registered new tserver with Master: ab608e4261bb447cbdabf47759ce35d2 (127.28.251.195:43227)
I20260808 03:58:51.395702 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 03:58:51.401998 13902 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 03:58:51.400394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.28.251.254:40305)
0808 03:58:51.400597 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:51.400603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:51.401038 (+ 435us) server_negotiation.cc:407] Connection header received
0808 03:58:51.401218 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:51.401222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:51.401278 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:51.401362 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:51.401775 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57710: BlockingRecv error: recv got EOF from 127.0.0.1:57710 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 03:58:51.402807 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 13694
I20260808 03:58:51.406551 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:51.404006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.28.251.193:36673)
0808 03:58:51.404666 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:51.404673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:51.404701 (+ 28us) server_negotiation.cc:407] Connection header received
0808 03:58:51.404762 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:51.404769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:51.404989 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:51.405119 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:51.405919 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:51.405926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:51.405938 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:51.405967 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:51.406013 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:51.406028 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:51.406137 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:51.406189 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 03:58:51.406274 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":368,"threads_started":1}
I20260808 03:58:51.407859 13890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:51.493134 13899 heartbeater.cc:503] Master 127.28.251.254:40305 was elected leader, sending a full tablet report...
I20260808 03:58:51.495121 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 13694
I20260808 03:58:51.516932 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 13904
I20260808 03:58:51.520530 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:51.517895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.28.251.194:42363)
0808 03:58:51.518440 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:51.518448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:51.518476 (+ 28us) server_negotiation.cc:407] Connection header received
0808 03:58:51.518542 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:51.518548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:51.518772 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:51.518901 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:51.519846 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:51.519852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:51.519864 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:51.519884 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:51.519926 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:51.519940 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:51.520051 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:51.520108 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 03:58:51.520207 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":235,"threads_started":1}
I20260808 03:58:51.521879 14097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:51.605242 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 13904
I20260808 03:58:51.633503 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 14111
I20260808 03:58:51.637715 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:51.634521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.28.251.195:43227)
0808 03:58:51.635180 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:51.635188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:51.635221 (+ 33us) server_negotiation.cc:407] Connection header received
0808 03:58:51.635289 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:51.635296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:51.635602 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:51.635758 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:51.636937 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:51.636945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:51.636956 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 03:58:51.636975 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:51.637016 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 03:58:51.637031 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:51.637192 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:51.637298 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 03:58:51.637407 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":334,"threads_started":1}
I20260808 03:58:51.639021 14315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:51.722287 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 14111
I20260808 03:58:51.747509 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 13551
I20260808 03:58:51.748759 13647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:51.866513 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 13551
2026-08-08T03:58:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2605 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:58:51 dist-test-slave-pww0 krb5kdc[14349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:58:51 dist-test-slave-pww0 krb5kdc[14349](info): set up 2 sockets
Aug 08 03:58:51 dist-test-slave-pww0 krb5kdc[14349](info): commencing operation
krb5kdc: starting...
W20260808 03:58:53.149485 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:58:53 dist-test-slave-pww0 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:58:53.470755 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:37255
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:37255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:53.825062 14374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:53.825414 14374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:53.825554 14374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:53.833092 14374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:58:53.833228 14374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:53.833309 14374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:53.833392 14374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:58:53.833441 14374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:58:53.843892 14374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:37255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:37255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:53.845355 14374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:53.846817 14374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:53.859266 14387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:53.859097 14395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:53.859134 14389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:53.860442 14374 server_base.cc:1034] running on GCE node
I20260808 03:58:53.861366 14374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:53.862916 14374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:53.864143 14374 hybrid_clock.cc:648] HybridClock initialized: now 1786161533864092 us; error 37 us; skew 500 ppm
I20260808 03:58:53.864522 14374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:53 dist-test-slave-pww0 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161533, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:53.867771 14374 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:58:53.869418 14374 webserver.cc:460] Webserver started at http://127.28.251.254:40037/ using document root <none> and password file <none>
I20260808 03:58:53.870151 14374 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:53.870280 14374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:53.870611 14374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:53.873834 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "bb07f0841ca64178a9c99ec1845e974f"
format_stamp: "Formatted at 2026-08-08 03:58:53 on dist-test-slave-pww0"
server_key: "1746ed4097e8119ee33a006cf9f28a8d"
server_key_iv: "64f5bd0501e8454b0a6e3a768144c8d4"
server_key_version: "encryptionkey@0"
I20260808 03:58:53.874675 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "bb07f0841ca64178a9c99ec1845e974f"
format_stamp: "Formatted at 2026-08-08 03:58:53 on dist-test-slave-pww0"
server_key: "1746ed4097e8119ee33a006cf9f28a8d"
server_key_iv: "64f5bd0501e8454b0a6e3a768144c8d4"
server_key_version: "encryptionkey@0"
I20260808 03:58:53.880395 14374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 03:58:53.884332 14405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:53.885746 14374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 03:58:53.885934 14374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "bb07f0841ca64178a9c99ec1845e974f"
format_stamp: "Formatted at 2026-08-08 03:58:53 on dist-test-slave-pww0"
server_key: "1746ed4097e8119ee33a006cf9f28a8d"
server_key_iv: "64f5bd0501e8454b0a6e3a768144c8d4"
server_key_version: "encryptionkey@0"
I20260808 03:58:53.886133 14374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:53.906499 14374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:53.911218 14374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:53.911554 14374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:53.929847 14374 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:37255
I20260808 03:58:53.929886 14487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:37255 every 8 connection(s)
I20260808 03:58:53.931330 14374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:58:53.932955 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 14374
I20260808 03:58:53.933172 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:58:53.933801 29679 external_mini_cluster.cc:1428] Setting key 3d6cc76abdc23bb4c9102a46d3d8a0a7
I20260808 03:58:53.935750 14490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 03:58:53 dist-test-slave-pww0 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:53.950177 14490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f: Bootstrap starting.
I20260808 03:58:53.954007 14490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:53.955495 14490 log.cc:824] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:53.958385 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:53.936186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.28.251.254:37255)
0808 03:58:53.936981 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:53.936994 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:53.937035 (+ 41us) server_negotiation.cc:407] Connection header received
0808 03:58:53.938679 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:53.938701 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:53.939100 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:53.939447 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:53.941349 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:53.942717 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:53.944491 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:53.945173 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:53.945312 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:53.950260 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:53.950269 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:53.950287 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:53.950372 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:53.953516 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:53.954456 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:53.954464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:53.954474 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:53.954653 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:53.955079 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:53.955082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:53.955084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:53.955877 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:53.956158 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:53.956852 (+ 694us) server_negotiation.cc:299] Negotiation successful
0808 03:58:53.957290 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":352,"threads_started":1}
I20260808 03:58:53.959002 14490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f: No bootstrap required, opened a new log
I20260808 03:58:53.967343 14490 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb07f0841ca64178a9c99ec1845e974f" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37255 } }
I20260808 03:58:53.967835 14490 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:53.967947 14490 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb07f0841ca64178a9c99ec1845e974f, State: Initialized, Role: FOLLOWER
I20260808 03:58:53.968581 14490 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb07f0841ca64178a9c99ec1845e974f" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37255 } }
I20260808 03:58:53.968804 14490 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:58:53.968942 14490 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:58:53.969107 14490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:53.971792 14490 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb07f0841ca64178a9c99ec1845e974f" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37255 } }
I20260808 03:58:53.972335 14490 leader_election.cc:302] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb07f0841ca64178a9c99ec1845e974f; no voters:
I20260808 03:58:53.972971 14490 leader_election.cc:288] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:58:53.973114 14496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:53.974489 14496 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [term 1 LEADER]: Becoming Leader. State: Replica: bb07f0841ca64178a9c99ec1845e974f, State: Running, Role: LEADER
I20260808 03:58:53.975123 14496 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb07f0841ca64178a9c99ec1845e974f" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37255 } }
I20260808 03:58:53.975353 14490 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:58:53.977945 14500 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb07f0841ca64178a9c99ec1845e974f. Latest consensus state: current_term: 1 leader_uuid: "bb07f0841ca64178a9c99ec1845e974f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb07f0841ca64178a9c99ec1845e974f" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37255 } } }
I20260808 03:58:53.978435 14500 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:53.979384 14499 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb07f0841ca64178a9c99ec1845e974f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb07f0841ca64178a9c99ec1845e974f" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 37255 } } }
I20260808 03:58:53.979791 14499 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f [sys.catalog]: This master's current role is: LEADER
I20260808 03:58:53.980907 14509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:58:53.986421 14509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:58:53.996778 14509 catalog_manager.cc:1347] Generated new cluster ID: 1692cc57aa3944d7bd6e6ec2096c4942
I20260808 03:58:53.996917 14509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:58:54.030550 14509 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:58:54.032603 14509 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:58:54.043807 14509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f: Generated new TSK 0
I20260808 03:58:54.045004 14509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:58:54.119611 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37255
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:54.483709 14531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:54.484053 14531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:54.484192 14531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:54.491861 14531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:54.492007 14531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:54.492161 14531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:58:54.502409 14531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:54.503926 14531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:54.505306 14531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:54.518656 14537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:54.518586 14541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:54.518632 14544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:54.519560 14531 server_base.cc:1034] running on GCE node
I20260808 03:58:54.520196 14531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:54.521196 14531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:54.522413 14531 hybrid_clock.cc:648] HybridClock initialized: now 1786161534522351 us; error 49 us; skew 500 ppm
I20260808 03:58:54.522758 14531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:54 dist-test-slave-pww0 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161534, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:54.526212 14531 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:58:54.527791 14531 webserver.cc:460] Webserver started at http://127.28.251.193:36503/ using document root <none> and password file <none>
I20260808 03:58:54.528496 14531 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:54.528604 14531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:54.528878 14531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:54.532068 14531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "2817f0479ce34e279c6fadebc5568f5d"
format_stamp: "Formatted at 2026-08-08 03:58:54 on dist-test-slave-pww0"
server_key: "07da9b1010e26fc5ecf6fd9477dc5099"
server_key_iv: "83a200cc7db51f90b454609b31664fb7"
server_key_version: "encryptionkey@0"
I20260808 03:58:54.532931 14531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "2817f0479ce34e279c6fadebc5568f5d"
format_stamp: "Formatted at 2026-08-08 03:58:54 on dist-test-slave-pww0"
server_key: "07da9b1010e26fc5ecf6fd9477dc5099"
server_key_iv: "83a200cc7db51f90b454609b31664fb7"
server_key_version: "encryptionkey@0"
I20260808 03:58:54.538601 14531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 03:58:54.542698 14554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:54.544302 14531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:54.544487 14531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "2817f0479ce34e279c6fadebc5568f5d"
format_stamp: "Formatted at 2026-08-08 03:58:54 on dist-test-slave-pww0"
server_key: "07da9b1010e26fc5ecf6fd9477dc5099"
server_key_iv: "83a200cc7db51f90b454609b31664fb7"
server_key_version: "encryptionkey@0"
I20260808 03:58:54.544697 14531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:54.570664 14531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:54.575255 14531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:54.575611 14531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:54.576720 14531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:54.578162 14531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:54.578239 14531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:54.578363 14531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:54.578413 14531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:54.612102 14531 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:33271
I20260808 03:58:54.612160 14740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:33271 every 8 connection(s)
I20260808 03:58:54.613530 14531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:58:54.614188 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 14531
I20260808 03:58:54.614423 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:58:54.615327 29679 external_mini_cluster.cc:1428] Setting key 2df0b13a3ac845efc6dcd7be5df67ab3
Aug 08 03:58:54 dist-test-slave-pww0 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161534, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:54.637055 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:54.616632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:55435 (local address 127.28.251.254:37255)
0808 03:58:54.617142 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:54.617148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:54.618301 (+ 1153us) server_negotiation.cc:407] Connection header received
0808 03:58:54.620301 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:54.620307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:54.620424 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:54.620542 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:54.622177 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:54.623144 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:54.625276 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:54.625809 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:54.625946 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:54.630443 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:54.630449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:54.630454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:54.630500 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:54.633517 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:54.634417 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:54.634422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:54.634424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:54.634572 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:54.635081 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:54.635086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:54.635088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:54.635398 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:54.635596 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:54.636407 (+ 811us) server_negotiation.cc:299] Negotiation successful
0808 03:58:54.636675 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":247,"threads_started":1}
I20260808 03:58:54.638156 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:54.617281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37255 (local address 127.28.251.193:55435)
0808 03:58:54.618118 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:54.618166 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:54.620013 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:54.620811 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:54.620820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:54.621365 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:54.621989 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:54.622007 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:54.623328 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:54.623334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:54.625078 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:54.625094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:54.626040 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:54.626046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:54.626322 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:54.627088 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:54.627116 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:54.630067 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:54.633714 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:54.633723 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:54.633739 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:54.634248 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:54.634699 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:54.634705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:54.634709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:54.634945 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:54.635565 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:54.635574 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:54.635975 (+ 401us) client_negotiation.cc:769] Sending connection context
0808 03:58:54.636246 (+ 271us) client_negotiation.cc:240] Negotiation successful
0808 03:58:54.636870 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":338,"threads_started":1}
I20260808 03:58:54.640046 14741 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37255
I20260808 03:58:54.640381 14741 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:54.641175 14741 heartbeater.cc:511] Master 127.28.251.254:37255 requested a full tablet report, sending...
I20260808 03:58:54.643782 14435 ts_manager.cc:194] Registered new tserver with Master: 2817f0479ce34e279c6fadebc5568f5d (127.28.251.193:33271)
I20260808 03:58:54.646003 14435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:55435
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:58:54.688002 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37255
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:55.048169 14750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:55.048655 14750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:55.048859 14750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:55.058195 14750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:55.058324 14750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:55.058451 14750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:58:55.068956 14750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:55.070429 14750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:55.071872 14750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:55.085861 14760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:55.085902 14765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:55.085824 14761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:55.086745 14750 server_base.cc:1034] running on GCE node
I20260808 03:58:55.087427 14750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:55.088506 14750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:55.089713 14750 hybrid_clock.cc:648] HybridClock initialized: now 1786161535089660 us; error 39 us; skew 500 ppm
I20260808 03:58:55.090057 14750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:55 dist-test-slave-pww0 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161535, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:55.093483 14750 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:58:55.095129 14750 webserver.cc:460] Webserver started at http://127.28.251.194:43219/ using document root <none> and password file <none>
I20260808 03:58:55.095927 14750 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:55.096063 14750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:55.096406 14750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:55.099646 14750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "46256364edda43fbbd5722bec4bed1b8"
format_stamp: "Formatted at 2026-08-08 03:58:55 on dist-test-slave-pww0"
server_key: "1d1d355f91da69465cec0e735a084753"
server_key_iv: "2114789712a3743d8f884d002b79558c"
server_key_version: "encryptionkey@0"
I20260808 03:58:55.100510 14750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "46256364edda43fbbd5722bec4bed1b8"
format_stamp: "Formatted at 2026-08-08 03:58:55 on dist-test-slave-pww0"
server_key: "1d1d355f91da69465cec0e735a084753"
server_key_iv: "2114789712a3743d8f884d002b79558c"
server_key_version: "encryptionkey@0"
I20260808 03:58:55.106140 14750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 03:58:55.109930 14777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:55.111388 14750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 03:58:55.111598 14750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "46256364edda43fbbd5722bec4bed1b8"
format_stamp: "Formatted at 2026-08-08 03:58:55 on dist-test-slave-pww0"
server_key: "1d1d355f91da69465cec0e735a084753"
server_key_iv: "2114789712a3743d8f884d002b79558c"
server_key_version: "encryptionkey@0"
I20260808 03:58:55.111817 14750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:55.126933 14750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:55.132942 14750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:55.133383 14750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:55.134791 14750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:55.136801 14750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:55.136946 14750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 03:58:55.137089 14750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:55.137158 14750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:55.169790 14750 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:38443
I20260808 03:58:55.169860 14959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:38443 every 8 connection(s)
I20260808 03:58:55.171196 14750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:58:55.176584 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 14750
I20260808 03:58:55.176797 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:58:55.177249 29679 external_mini_cluster.cc:1428] Setting key 37371f75bbf0436c76c6245970226d79
Aug 08 03:58:55 dist-test-slave-pww0 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161535, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:55.193593 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.173867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:52379 (local address 127.28.251.254:37255)
0808 03:58:55.174373 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:55.174379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:55.175510 (+ 1131us) server_negotiation.cc:407] Connection header received
0808 03:58:55.177647 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:55.177653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:55.177759 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:55.177886 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:55.180034 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.181040 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.182638 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.183267 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.183400 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.187090 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:55.187096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:55.187101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:55.187143 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:55.190279 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:55.191135 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:55.191141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:55.191144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:55.191312 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:55.191756 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:55.191760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:55.191763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:55.192123 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:55.192262 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:55.193014 (+ 752us) server_negotiation.cc:299] Negotiation successful
0808 03:58:55.193241 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":240,"threads_started":1}
I20260808 03:58:55.194638 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.174490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37255 (local address 127.28.251.194:52379)
0808 03:58:55.175272 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:55.175347 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:55.177320 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:55.178170 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:55.178180 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:55.178989 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:55.179844 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.179872 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.181179 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.181183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.182485 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.182499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.183525 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.183533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.183753 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.184444 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:55.184469 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:55.186823 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:55.190475 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:55.190483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:55.190499 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:55.191005 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:55.191406 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:55.191409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:55.191412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:55.191646 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:55.192238 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:55.192245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:55.192608 (+ 363us) client_negotiation.cc:769] Sending connection context
0808 03:58:55.192864 (+ 256us) client_negotiation.cc:240] Negotiation successful
0808 03:58:55.193325 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":282,"threads_started":1}
I20260808 03:58:55.196159 14960 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37255
I20260808 03:58:55.196487 14960 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:55.197278 14960 heartbeater.cc:511] Master 127.28.251.254:37255 requested a full tablet report, sending...
I20260808 03:58:55.198815 14435 ts_manager.cc:194] Registered new tserver with Master: 46256364edda43fbbd5722bec4bed1b8 (127.28.251.194:38443)
I20260808 03:58:55.199704 14435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:52379
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:58:55.246536 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37255
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:58:55.603559 14967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:58:55.603893 14967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:58:55.604038 14967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:58:55.611733 14967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:58:55.611904 14967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:58:55.612077 14967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:58:55.622489 14967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:37255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:58:55.623929 14967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:58:55.625278 14967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:58:55.638919 14975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:55.638989 14976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:58:55.638957 14978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:58:55.639761 14967 server_base.cc:1034] running on GCE node
I20260808 03:58:55.640455 14967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:58:55.641470 14967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:58:55.642719 14967 hybrid_clock.cc:648] HybridClock initialized: now 1786161535642662 us; error 45 us; skew 500 ppm
I20260808 03:58:55.643080 14967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:58:55 dist-test-slave-pww0 krb5kdc[14349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161535, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:58:55.649427 14967 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:58:55.649801 14741 heartbeater.cc:503] Master 127.28.251.254:37255 was elected leader, sending a full tablet report...
I20260808 03:58:55.651443 14967 webserver.cc:460] Webserver started at http://127.28.251.195:32871/ using document root <none> and password file <none>
I20260808 03:58:55.652272 14967 fs_manager.cc:359] Metadata directory not provided
I20260808 03:58:55.652432 14967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:58:55.652747 14967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 03:58:55.655969 14967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "ed4b9cba7e6742c5bb2e1778198605b4"
format_stamp: "Formatted at 2026-08-08 03:58:55 on dist-test-slave-pww0"
server_key: "cd3e88aad3da943d6c1f9b97fc45f015"
server_key_iv: "ef0ff93644598b892ac0e127c30297fc"
server_key_version: "encryptionkey@0"
I20260808 03:58:55.656828 14967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "ed4b9cba7e6742c5bb2e1778198605b4"
format_stamp: "Formatted at 2026-08-08 03:58:55 on dist-test-slave-pww0"
server_key: "cd3e88aad3da943d6c1f9b97fc45f015"
server_key_iv: "ef0ff93644598b892ac0e127c30297fc"
server_key_version: "encryptionkey@0"
I20260808 03:58:55.662832 14967 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260808 03:58:55.666884 14990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:55.668517 14967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 03:58:55.668701 14967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "ed4b9cba7e6742c5bb2e1778198605b4"
format_stamp: "Formatted at 2026-08-08 03:58:55 on dist-test-slave-pww0"
server_key: "cd3e88aad3da943d6c1f9b97fc45f015"
server_key_iv: "ef0ff93644598b892ac0e127c30297fc"
server_key_version: "encryptionkey@0"
I20260808 03:58:55.668923 14967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:58:55.714129 14967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:58:55.719049 14967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:58:55.719363 14967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:58:55.720455 14967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:58:55.721966 14967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:58:55.722043 14967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:55.722160 14967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:58:55.722214 14967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:58:55.755577 14967 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:42559
I20260808 03:58:55.755617 15181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:42559 every 8 connection(s)
I20260808 03:58:55.756940 14967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:58:55.761111 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 14967
I20260808 03:58:55.761349 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:58:55.761930 29679 external_mini_cluster.cc:1428] Setting key e714a280f9f0be174635b1bdd66fda3f
Aug 08 03:58:55 dist-test-slave-pww0 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161535, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:58:55.779512 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.759552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:43931 (local address 127.28.251.254:37255)
0808 03:58:55.760053 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:55.760058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:55.761179 (+ 1121us) server_negotiation.cc:407] Connection header received
0808 03:58:55.763743 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:55.763749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:55.763855 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:55.763982 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:55.766129 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.766918 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.768641 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.769077 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.769185 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.772994 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:55.772999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:55.773003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:55.773043 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:55.775962 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:55.776926 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:55.776931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:55.776933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:55.777087 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:55.777568 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:55.777573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:55.777576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:55.777933 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:55.778060 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:55.778963 (+ 903us) server_negotiation.cc:299] Negotiation successful
0808 03:58:55.779147 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
I20260808 03:58:55.780577 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.760072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:37255 (local address 127.28.251.195:43931)
0808 03:58:55.760945 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:55.761012 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:55.763376 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:55.764252 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:55.764265 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:55.765066 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:58:55.765942 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.765966 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.767072 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.767076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.768473 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.768487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.769302 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.769308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.769549 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.770261 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:58:55.770286 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:58:55.772727 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:58:55.776146 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:55.776155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:55.776171 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:55.776778 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:55.777203 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:58:55.777208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:58:55.777211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:58:55.777438 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:58:55.778090 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:58:55.778099 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:58:55.778558 (+ 459us) client_negotiation.cc:769] Sending connection context
0808 03:58:55.778806 (+ 248us) client_negotiation.cc:240] Negotiation successful
0808 03:58:55.779300 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":299,"threads_started":1}
I20260808 03:58:55.782028 15182 heartbeater.cc:347] Connected to a master server at 127.28.251.254:37255
I20260808 03:58:55.782366 15182 heartbeater.cc:464] Registering TS with master...
I20260808 03:58:55.783180 15182 heartbeater.cc:511] Master 127.28.251.254:37255 requested a full tablet report, sending...
I20260808 03:58:55.784778 14435 ts_manager.cc:194] Registered new tserver with Master: ed4b9cba7e6742c5bb2e1778198605b4 (127.28.251.195:42559)
I20260808 03:58:55.785641 14435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:43931
I20260808 03:58:55.790933 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 03:58:55.807926 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.795712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.28.251.254:37255)
0808 03:58:55.795890 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:55.795895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:55.796408 (+ 513us) server_negotiation.cc:407] Connection header received
0808 03:58:55.796519 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:55.796523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:55.796593 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:55.796683 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:55.798214 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.798983 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.800281 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.800723 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.800823 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.803088 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:55.803092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:55.803096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:55.803131 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:55.805557 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:55.806205 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:55.806210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:55.806213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:55.806339 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:55.806629 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:55.806632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:55.806634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:55.806977 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:55.807159 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:55.807445 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 03:58:55.807669 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 03:58:55.812263 14435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:58:55.819648 14435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:58:55.844075 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.833198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.28.251.195:42559)
0808 03:58:55.833855 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:55.833866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:55.833903 (+ 37us) server_negotiation.cc:407] Connection header received
0808 03:58:55.834315 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:55.834325 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:55.834631 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:55.834933 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:55.836079 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.837862 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.840814 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.842314 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.842513 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.842713 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:55.843505 (+ 792us) server_negotiation.cc:299] Negotiation successful
0808 03:58:55.843697 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":300,"threads_started":1}
I20260808 03:58:55.844161 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.833024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:42559 (local address 127.0.0.1:41344)
0808 03:58:55.833776 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:55.833833 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:55.834007 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:55.835290 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:55.835298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:55.835330 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:55.835836 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.835850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.838042 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.838049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.840657 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.840669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.842861 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.842870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.843140 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.843254 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 03:58:55.843406 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 03:58:55.843647 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":29,"thread_start_us":316,"threads_started":1}
I20260808 03:58:55.847307 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.835029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47600 (local address 127.28.251.194:38443)
0808 03:58:55.835959 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:55.835968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:55.839284 (+ 3316us) server_negotiation.cc:407] Connection header received
0808 03:58:55.839399 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:55.839408 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:55.839716 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:55.840188 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:55.840856 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.842097 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.844705 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.846170 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.846342 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.846522 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:55.846833 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 03:58:55.847007 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":621,"threads_started":1}
I20260808 03:58:55.847373 15214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.835300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:38443 (local address 127.0.0.1:47600)
0808 03:58:55.838597 (+ 3297us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:55.838614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:55.838738 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:55.839931 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:55.839936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:55.839962 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:55.840628 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.840639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.842256 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.842261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.844524 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.844541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.846499 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.846506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.846658 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.846684 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 03:58:55.846766 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 03:58:55.846903 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3167,"mutex_wait_us":127,"spinlock_wait_cycles":4864,"thread_start_us":2991,"threads_started":1}
I20260808 03:58:55.848544 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.834517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.28.251.193:33271)
0808 03:58:55.839353 (+ 4836us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:55.839362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:55.839393 (+ 31us) server_negotiation.cc:407] Connection header received
0808 03:58:55.839519 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:55.839528 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:55.839704 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:55.839905 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:55.840883 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.842430 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.845038 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.846903 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.847836 (+ 933us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.847995 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:55.848055 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 03:58:55.848216 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4639,"thread_start_us":4454,"threads_started":1}
I20260808 03:58:55.849022 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.834143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33271 (local address 127.0.0.1:49404)
0808 03:58:55.834675 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:55.834694 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:55.834816 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:55.839902 (+ 5086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:55.839906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:55.839932 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:55.840625 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.840639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.842588 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.842593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.844876 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.844891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.847155 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.847162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.847340 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.847379 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 03:58:55.847451 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 03:58:55.847616 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":5,"thread_start_us":261,"threads_started":1}
I20260808 03:58:55.849000 15067 tablet_service.cc:1467] Processing CreateTablet for tablet fcfb0e7aeb914e808e94cc923ddaed6b (DEFAULT_TABLE table=test-table [id=d67c5732e4454f8d913d9d1ee67e8ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:55.850888 15067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfb0e7aeb914e808e94cc923ddaed6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:55.852051 14854 tablet_service.cc:1467] Processing CreateTablet for tablet fcfb0e7aeb914e808e94cc923ddaed6b (DEFAULT_TABLE table=test-table [id=d67c5732e4454f8d913d9d1ee67e8ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:55.852435 14634 tablet_service.cc:1467] Processing CreateTablet for tablet fcfb0e7aeb914e808e94cc923ddaed6b (DEFAULT_TABLE table=test-table [id=d67c5732e4454f8d913d9d1ee67e8ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:58:55.853621 14854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfb0e7aeb914e808e94cc923ddaed6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:55.853643 14634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcfb0e7aeb914e808e94cc923ddaed6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:58:55.865978 15216 tablet_bootstrap.cc:492] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d: Bootstrap starting.
I20260808 03:58:55.868710 15218 tablet_bootstrap.cc:492] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: Bootstrap starting.
I20260808 03:58:55.870438 15216 tablet_bootstrap.cc:654] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:55.871965 15216 log.cc:824] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:55.872331 15217 tablet_bootstrap.cc:492] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: Bootstrap starting.
I20260808 03:58:55.872823 15218 tablet_bootstrap.cc:654] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:55.874267 15218 log.cc:824] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:55.876269 15217 tablet_bootstrap.cc:654] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: Neither blocks nor log segments found. Creating new log.
I20260808 03:58:55.876371 15216 tablet_bootstrap.cc:492] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d: No bootstrap required, opened a new log
I20260808 03:58:55.876747 15216 ts_tablet_manager.cc:1397] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260808 03:58:55.877395 15217 log.cc:824] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: Log is configured to *not* fsync() on all Append() calls
I20260808 03:58:55.877856 15218 tablet_bootstrap.cc:492] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: No bootstrap required, opened a new log
I20260808 03:58:55.878275 15218 ts_tablet_manager.cc:1397] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260808 03:58:55.880848 15217 tablet_bootstrap.cc:492] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: No bootstrap required, opened a new log
I20260808 03:58:55.881148 15217 ts_tablet_manager.cc:1397] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 03:58:55.887760 15216 raft_consensus.cc:348] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 42559 } } peers { permanent_uuid: "2817f0479ce34e279c6fadebc5568f5d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33271 } } peers { permanent_uuid: "46256364edda43fbbd5722bec4bed1b8" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38443 } }
I20260808 03:58:55.888468 15216 raft_consensus.cc:374] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:55.888339 15217 raft_consensus.cc:348] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 42559 } } peers { permanent_uuid: "2817f0479ce34e279c6fadebc5568f5d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33271 } } peers { permanent_uuid: "46256364edda43fbbd5722bec4bed1b8" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38443 } }
I20260808 03:58:55.888662 15216 raft_consensus.cc:729] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2817f0479ce34e279c6fadebc5568f5d, State: Initialized, Role: FOLLOWER
I20260808 03:58:55.888700 15217 raft_consensus.cc:374] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:55.888844 15217 raft_consensus.cc:729] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4b9cba7e6742c5bb2e1778198605b4, State: Initialized, Role: FOLLOWER
I20260808 03:58:55.889101 15218 raft_consensus.cc:348] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 42559 } } peers { permanent_uuid: "2817f0479ce34e279c6fadebc5568f5d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33271 } } peers { permanent_uuid: "46256364edda43fbbd5722bec4bed1b8" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38443 } }
I20260808 03:58:55.889501 15217 consensus_queue.cc:260] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 42559 } } peers { permanent_uuid: "2817f0479ce34e279c6fadebc5568f5d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33271 } } peers { permanent_uuid: "46256364edda43fbbd5722bec4bed1b8" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38443 } }
I20260808 03:58:55.889575 15216 consensus_queue.cc:260] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 42559 } } peers { permanent_uuid: "2817f0479ce34e279c6fadebc5568f5d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33271 } } peers { permanent_uuid: "46256364edda43fbbd5722bec4bed1b8" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38443 } }
I20260808 03:58:55.889766 15218 raft_consensus.cc:374] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:58:55.889972 15218 raft_consensus.cc:729] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46256364edda43fbbd5722bec4bed1b8, State: Initialized, Role: FOLLOWER
I20260808 03:58:55.890808 15218 consensus_queue.cc:260] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 42559 } } peers { permanent_uuid: "2817f0479ce34e279c6fadebc5568f5d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33271 } } peers { permanent_uuid: "46256364edda43fbbd5722bec4bed1b8" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38443 } }
I20260808 03:58:55.892807 15182 heartbeater.cc:503] Master 127.28.251.254:37255 was elected leader, sending a full tablet report...
I20260808 03:58:55.893162 15216 ts_tablet_manager.cc:1428] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260808 03:58:55.894345 14960 heartbeater.cc:503] Master 127.28.251.254:37255 was elected leader, sending a full tablet report...
I20260808 03:58:55.895365 15217 ts_tablet_manager.cc:1428] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: Time spent starting tablet: real 0.014s user 0.008s sys 0.002s
I20260808 03:58:55.896472 15218 ts_tablet_manager.cc:1428] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: Time spent starting tablet: real 0.018s user 0.007s sys 0.008s
W20260808 03:58:55.923308 14961 tablet.cc:2369] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:58:55.952971 15226 raft_consensus.cc:482] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:58:55.953316 15226 raft_consensus.cc:504] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 42559 } } peers { permanent_uuid: "2817f0479ce34e279c6fadebc5568f5d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33271 } } peers { permanent_uuid: "46256364edda43fbbd5722bec4bed1b8" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38443 } }
I20260808 03:58:55.955175 15226 leader_election.cc:288] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed4b9cba7e6742c5bb2e1778198605b4 (127.28.251.195:42559), 46256364edda43fbbd5722bec4bed1b8 (127.28.251.194:38443)
I20260808 03:58:55.962560 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.955616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:35159 (local address 127.28.251.195:42559)
0808 03:58:55.955820 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:55.955827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:55.956383 (+ 556us) server_negotiation.cc:407] Connection header received
0808 03:58:55.956646 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:55.956652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:55.956740 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:55.956840 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:55.957509 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.958450 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.960634 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.961487 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.961625 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.961684 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:55.962131 (+ 447us) server_negotiation.cc:299] Negotiation successful
0808 03:58:55.962278 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 03:58:55.962663 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.955569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:42559 (local address 127.28.251.193:35159)
0808 03:58:55.956248 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:55.956265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:55.956440 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:55.956889 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:55.956898 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:55.956931 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:55.957336 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.957349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.958591 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.958597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.960463 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.960477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.961729 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.961736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.961946 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.961985 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 03:58:55.962075 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 03:58:55.962198 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":104,"thread_start_us":354,"threads_started":1}
I20260808 03:58:55.962702 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.956634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:50217 (local address 127.28.251.194:38443)
0808 03:58:55.956808 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:55.956814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:55.957189 (+ 375us) server_negotiation.cc:407] Connection header received
0808 03:58:55.957327 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:55.957332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:55.957414 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:55.957498 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:58:55.958090 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.958951 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.960813 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.961711 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.961875 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.961951 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:55.962295 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 03:58:55.962466 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 03:58:55.962781 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:55.956440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:38443 (local address 127.28.251.193:50217)
0808 03:58:55.957078 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0808 03:58:55.957096 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:58:55.957204 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:58:55.957535 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:58:55.957539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:58:55.957566 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:58:55.957954 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.957968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.959091 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.959095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.960620 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:58:55.960636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:58:55.961944 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:58:55.961949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:58:55.962116 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:55.962149 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 03:58:55.962241 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 03:58:55.962372 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":3,"thread_start_us":370,"threads_started":1}
I20260808 03:58:55.963950 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfb0e7aeb914e808e94cc923ddaed6b" candidate_uuid: "2817f0479ce34e279c6fadebc5568f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" is_pre_election: true
I20260808 03:58:55.963954 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfb0e7aeb914e808e94cc923ddaed6b" candidate_uuid: "2817f0479ce34e279c6fadebc5568f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46256364edda43fbbd5722bec4bed1b8" is_pre_election: true
I20260808 03:58:55.964541 14885 raft_consensus.cc:2454] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2817f0479ce34e279c6fadebc5568f5d in term 0.
I20260808 03:58:55.964556 15105 raft_consensus.cc:2454] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2817f0479ce34e279c6fadebc5568f5d in term 0.
I20260808 03:58:55.965294 14558 leader_election.cc:302] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2817f0479ce34e279c6fadebc5568f5d, 46256364edda43fbbd5722bec4bed1b8; no voters:
I20260808 03:58:55.965693 15226 raft_consensus.cc:2781] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:58:55.965860 15226 raft_consensus.cc:482] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:58:55.965976 15226 raft_consensus.cc:3037] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:55.968326 15226 raft_consensus.cc:504] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 42559 } } peers { permanent_uuid: "2817f0479ce34e279c6fadebc5568f5d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33271 } } peers { permanent_uuid: "46256364edda43fbbd5722bec4bed1b8" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38443 } }
I20260808 03:58:55.968978 15226 leader_election.cc:288] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [CANDIDATE]: Term 1 election: Requested vote from peers ed4b9cba7e6742c5bb2e1778198605b4 (127.28.251.195:42559), 46256364edda43fbbd5722bec4bed1b8 (127.28.251.194:38443)
I20260808 03:58:55.969539 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfb0e7aeb914e808e94cc923ddaed6b" candidate_uuid: "2817f0479ce34e279c6fadebc5568f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4b9cba7e6742c5bb2e1778198605b4"
I20260808 03:58:55.969605 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcfb0e7aeb914e808e94cc923ddaed6b" candidate_uuid: "2817f0479ce34e279c6fadebc5568f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46256364edda43fbbd5722bec4bed1b8"
I20260808 03:58:55.969748 15105 raft_consensus.cc:3037] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:55.969812 14885 raft_consensus.cc:3037] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:58:55.972185 14885 raft_consensus.cc:2454] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2817f0479ce34e279c6fadebc5568f5d in term 1.
I20260808 03:58:55.972191 15105 raft_consensus.cc:2454] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2817f0479ce34e279c6fadebc5568f5d in term 1.
I20260808 03:58:55.972666 14558 leader_election.cc:302] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2817f0479ce34e279c6fadebc5568f5d, 46256364edda43fbbd5722bec4bed1b8; no voters:
I20260808 03:58:55.973007 15226 raft_consensus.cc:2781] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:58:55.973505 15226 raft_consensus.cc:686] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 1 LEADER]: Becoming Leader. State: Replica: 2817f0479ce34e279c6fadebc5568f5d, State: Running, Role: LEADER
I20260808 03:58:55.974079 15226 consensus_queue.cc:237] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 42559 } } peers { permanent_uuid: "2817f0479ce34e279c6fadebc5568f5d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33271 } } peers { permanent_uuid: "46256364edda43fbbd5722bec4bed1b8" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38443 } }
I20260808 03:58:55.978821 14434 catalog_manager.cc:5495] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2817f0479ce34e279c6fadebc5568f5d (127.28.251.193). New cstate: current_term: 1 leader_uuid: "2817f0479ce34e279c6fadebc5568f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 42559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2817f0479ce34e279c6fadebc5568f5d" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46256364edda43fbbd5722bec4bed1b8" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20260808 03:58:56.007582 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:56.002441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.28.251.193:33271)
0808 03:58:56.002690 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:56.002695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:56.002715 (+ 20us) server_negotiation.cc:407] Connection header received
0808 03:58:56.002887 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:56.002892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:56.002976 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:56.003106 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:58:56.003444 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:56.004209 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:56.005504 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:56.005939 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:56.006039 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:56.006493 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:58:56.007020 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:58:56.007132 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:56.007179 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 03:58:56.007253 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260808 03:58:56.008710 15186 tablet.cc:2369] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 03:58:56.017102 14885 raft_consensus.cc:1261] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8 [term 1 FOLLOWER]: Refusing update from remote peer 2817f0479ce34e279c6fadebc5568f5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:56.017102 15105 raft_consensus.cc:1261] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4 [term 1 FOLLOWER]: Refusing update from remote peer 2817f0479ce34e279c6fadebc5568f5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:58:56.018152 15232 consensus_queue.cc:1035] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed4b9cba7e6742c5bb2e1778198605b4" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 42559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:56.018656 15226 consensus_queue.cc:1035] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "46256364edda43fbbd5722bec4bed1b8" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:58:56.047780 14435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:56.048067 14435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:58:56.051805 14435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb07f0841ca64178a9c99ec1845e974f: Sending DeleteTablet for 3 replicas of tablet fcfb0e7aeb914e808e94cc923ddaed6b
I20260808 03:58:56.053136 15067 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfb0e7aeb914e808e94cc923ddaed6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:56 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:41344
I20260808 03:58:56.053274 14854 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfb0e7aeb914e808e94cc923ddaed6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:56 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:47600
I20260808 03:58:56.054353 14634 tablet_service.cc:1514] Processing DeleteTablet for tablet fcfb0e7aeb914e808e94cc923ddaed6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:58:56 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:49404
I20260808 03:58:56.059908 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 14531
I20260808 03:58:56.066414 15253 tablet_replica.cc:330] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: stopping tablet replica
I20260808 03:58:56.066993 15253 raft_consensus.cc:2229] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:56.067636 15253 raft_consensus.cc:2258] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:56.068643 15255 tablet_replica.cc:330] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: stopping tablet replica
I20260808 03:58:56.069155 15255 raft_consensus.cc:2229] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:58:56.069682 15255 raft_consensus.cc:2258] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:56.072314 15255 ts_tablet_manager.cc:1905] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:56.073065 15253 ts_tablet_manager.cc:1905] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:58:56.080586 15255 ts_tablet_manager.cc:1918] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:56.080598 15253 ts_tablet_manager.cc:1918] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:56.080797 15253 log.cc:1196] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/fcfb0e7aeb914e808e94cc923ddaed6b
I20260808 03:58:56.080983 15255 log.cc:1196] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/fcfb0e7aeb914e808e94cc923ddaed6b
I20260808 03:58:56.081465 15253 ts_tablet_manager.cc:1939] T fcfb0e7aeb914e808e94cc923ddaed6b P ed4b9cba7e6742c5bb2e1778198605b4: Deleting consensus metadata
I20260808 03:58:56.081686 15255 ts_tablet_manager.cc:1939] T fcfb0e7aeb914e808e94cc923ddaed6b P 46256364edda43fbbd5722bec4bed1b8: Deleting consensus metadata
I20260808 03:58:56.083711 14412 catalog_manager.cc:4841] TS ed4b9cba7e6742c5bb2e1778198605b4 (127.28.251.195:42559): tablet fcfb0e7aeb914e808e94cc923ddaed6b (table test-table [id=d67c5732e4454f8d913d9d1ee67e8ad7]) successfully deleted
I20260808 03:58:56.084110 14412 catalog_manager.cc:4841] TS 46256364edda43fbbd5722bec4bed1b8 (127.28.251.194:38443): tablet fcfb0e7aeb914e808e94cc923ddaed6b (table test-table [id=d67c5732e4454f8d913d9d1ee67e8ad7]) successfully deleted
I20260808 03:58:56.086822 15256 tablet_replica.cc:330] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d: stopping tablet replica
I20260808 03:58:56.087371 15256 raft_consensus.cc:2229] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:58:56.088274 15256 raft_consensus.cc:2258] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:58:56.090441 15256 ts_tablet_manager.cc:1905] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 03:58:56 dist-test-slave-pww0 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:58:56.097466 15256 ts_tablet_manager.cc:1918] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:58:56.097689 15256 log.cc:1196] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/fcfb0e7aeb914e808e94cc923ddaed6b
I20260808 03:58:56.098313 15256 ts_tablet_manager.cc:1939] T fcfb0e7aeb914e808e94cc923ddaed6b P 2817f0479ce34e279c6fadebc5568f5d: Deleting consensus metadata
I20260808 03:58:56.100521 14412 catalog_manager.cc:4841] TS 2817f0479ce34e279c6fadebc5568f5d (127.28.251.193:33271): tablet fcfb0e7aeb914e808e94cc923ddaed6b (table test-table [id=d67c5732e4454f8d913d9d1ee67e8ad7]) successfully deleted
I20260808 03:58:56.104905 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:56.061770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.28.251.193:33271)
0808 03:58:56.061998 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:56.062006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:56.086040 (+ 24034us) server_negotiation.cc:407] Connection header received
0808 03:58:56.086294 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:56.086301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:56.086444 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:56.086593 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:56.088249 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:56.089555 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:56.091410 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:56.092248 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:56.092398 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:56.097506 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:56.097522 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:56.097541 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:56.097607 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:56.101393 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:56.102318 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:56.102349 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:56.102353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:56.102566 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:56.103084 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:56.103090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:56.103094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:56.103639 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:56.103981 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:56.104201 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 03:58:56.104469 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 03:58:56.105718 14732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:56.217828 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 14531
I20260808 03:58:56.252647 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 14750
Aug 08 03:58:56 dist-test-slave-pww0 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 03:58:56.273233 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:56.253594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.28.251.194:38443)
0808 03:58:56.253795 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:56.253800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:56.253823 (+ 23us) server_negotiation.cc:407] Connection header received
0808 03:58:56.253950 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:56.253954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:56.254022 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:56.254123 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:56.255537 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:56.256700 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:56.258619 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:56.259275 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:56.259369 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:56.267182 (+ 7813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:56.267206 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:56.267223 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:56.267270 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:56.270185 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:56.271087 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:56.271095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:56.271097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:56.271231 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:56.271683 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:56.271686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:56.271688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:56.272095 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:56.272427 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:56.272731 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 03:58:56.272921 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 03:58:56.273988 14955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:56.368634 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 14750
I20260808 03:58:56.397811 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 14967
Aug 08 03:58:56 dist-test-slave-pww0 krb5kdc[14349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 03:58:56.412716 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:58:56.398661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.28.251.195:42559)
0808 03:58:56.398862 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 03:58:56.398867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:58:56.398888 (+ 21us) server_negotiation.cc:407] Connection header received
0808 03:58:56.399047 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:58:56.399051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:58:56.399119 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:58:56.399206 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:58:56.400495 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:56.401232 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:56.402475 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:58:56.402868 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:58:56.402960 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:58:56.407326 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:58:56.407346 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:58:56.407364 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:58:56.407405 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:58:56.410122 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:56.410832 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:56.410852 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:56.410854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:56.411010 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:58:56.411302 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:58:56.411305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:58:56.411307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:58:56.411701 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:58:56.411950 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 03:58:56.412202 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 03:58:56.412381 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 03:58:56.413391 15174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:56.509081 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 14967
I20260808 03:58:56.532637 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 14374
I20260808 03:58:56.533943 14482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:58:56.646406 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 14374
2026-08-08T03:58:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4779 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:58:56 dist-test-slave-pww0 krb5kdc[15269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:58:56 dist-test-slave-pww0 krb5kdc[15269](info): set up 2 sockets
Aug 08 03:58:56 dist-test-slave-pww0 krb5kdc[15269](info): commencing operation
krb5kdc: starting...
W20260808 03:58:57.934223 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:58:58 dist-test-slave-pww0 krb5kdc[15269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:58:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/rangeradmin_127.28.251.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/rangeradmin_127.28.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/rangerlookup_127.28.251.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/rangerlookup_127.28.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/HTTP_127.28.251.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/HTTP_127.28.251.212@KRBTEST.COM.keytab.
I20260808 03:58:58.392760 29679 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhSn85p/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/postgres -l logfile start
2026-08-08 03:59:00.891 UTC [15317] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 03:59:00.891 UTC [15317] LOG: listening on IPv4 address "127.28.251.212", port 48133
2026-08-08 03:59:00.895 UTC [15317] LOG: listening on Unix socket "/tmp/.s.PGSQL.48133"
2026-08-08 03:59:00.916 UTC [15319] LOG: database system was shut down at 2026-08-08 03:58:59 UTC
2026-08-08 03:59:00.920 UTC [15317] LOG: database system is ready to accept connections
I20260808 03:59:02.056456 29679 mini_postgres.cc:96] Postgres bound to 48133
2026-08-08 03:59:02.086 UTC [15344] FATAL: database "slave" does not exist
127.28.251.212:48133 - accepting connections
I20260808 03:59:02.087554 29679 mini_ranger.cc:161] Starting Ranger...
I20260808 03:59:02.125432 29679 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 03:59:02.593072 29679 mini_ranger.cc:87] Created ranger Postgres database
I20260808 03:59:02.593201 29679 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-admin
2026-08-08 03:59:02,944 [I] DB FLAVOR :POSTGRES
2026-08-08 03:59:02,944 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 03:59:02,945 [I] Checking connection..
2026-08-08 03:59:02,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 03:59:03,215 [I] Checking connection passed.
2026-08-08 03:59:03,215 [I] --------- Verifying version history table ---------
2026-08-08 03:59:03,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 03:59:03,460 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 03:59:03,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 03:59:03,738 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 03:59:03,739 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 03:59:03,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 03:59:04.007 UTC [15436] WARNING: there is no transaction in progress
2026-08-08 03:59:04,015 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 03:59:04,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 03:59:04,261 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 03:59:04,261 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 03:59:04,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 03:59:04,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 03:59:04,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pww0.c.gcp-upstream.internal','N') ;"
2026-08-08 03:59:05,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 03:59:05,318 [I] Table x_portal_user does not exist in database ranger
2026-08-08 03:59:05,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 03:59:05,575 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 03:59:05,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 03:59:05,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 03:59:06,104 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 03:59:06,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 03:59:07.107 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.124 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.141 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.157 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.174 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.192 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.251 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.259 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.274 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.284 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.295 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.309 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.320 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.327 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.335 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:07.342 UTC [15634] WARNING: there is no transaction in progress
2026-08-08 03:59:08,038 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 03:59:08,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 03:59:08,306 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 03:59:08,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 03:59:08,582 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 03:59:08,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 03:59:08,857 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 03:59:08,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 03:59:09,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 03:59:09,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pww0.c.gcp-upstream.internal';"
2026-08-08 03:59:09,646 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 03:59:09,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 03:59:09,894 [I] DB_PATCHES have already been applied
I20260808 03:59:09.900360 29679 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhSn85p/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 03:59:09.900461 29679 mini_ranger.cc:193] Using host: 127.28.251.212
I20260808 03:59:09.925348 29679 mini_ranger.cc:227] Ranger admin URL: http://127.28.251.212:41637
Aug 08, 2026 3:59:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:59:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:59:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 3:59:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 3:59:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 3:59:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 3:59:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 03:59:11.252825 29679 mini_ranger.cc:160] Time spent starting Ranger: real 9.165s user 0.004s sys 0.080s
Aug 08, 2026 3:59:32 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 3:59:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 08, 2026 3:59:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 3:59:32 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 3:59:33 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 03:59:34.165896 29679 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/rangerkms_127.28.251.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/rangerkms_127.28.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.251.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.251.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.251.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/HTTP_127.28.251.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.251.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/HTTP_127.28.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 03:59:34 dist-test-slave-pww0 krb5kdc[15269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161574, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 03:59:34.409075 29679 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 03:59:34.440204 29679 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 03:59:35.309437 29679 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 03:59:35.309618 29679 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-kms
I20260808 03:59:35.309772 29679 mini_ranger_kms.cc:227] Using postgres at 127.28.251.212:48133
2026-08-08 03:59:35,859 [I] DB FLAVOR :POSTGRES
2026-08-08 03:59:35,859 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 03:59:35,859 [I] Checking connection
2026-08-08 03:59:35,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 03:59:36,120 [I] connection success
2026-08-08 03:59:36,120 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 03:59:36,120 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 03:59:36,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 03:59:36,404 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 03:59:36,404 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 03:59:36,404 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 03:59:36,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:48133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 03:59:36,685 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 03:59:37.263221 29679 mini_ranger_kms.cc:310] Created kms service
I20260808 03:59:37.496224 29679 mini_ranger_kms.cc:326] Created kudu user
I20260808 03:59:37.546873 29679 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 03:59:37 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161574, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 03:59:37.985297 29679 mini_ranger_kms.cc:384] Added ranger policy
I20260808 03:59:37.986613 29679 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 03:59:37.986683 29679 mini_ranger_kms.cc:240] Using host: 127.28.251.212
I20260808 03:59:38.025743 29679 mini_ranger_kms.cc:276] Ranger KMS PID: 15929
I20260808 03:59:38.025938 29679 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.251.212:35305
Aug 08, 2026 3:59:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:59:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:59:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 3:59:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 3:59:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 3:59:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 03:59:40 dist-test-slave-pww0 krb5kdc[15269](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786161580, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.251.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:59:40.644369 29679 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.235s user 0.178s sys 0.436s
I20260808 03:59:40.644635 29679 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 03:59:40.644850 29679 mini_ranger_kms.cc:401] 127.28.251.212:35305/kms/v1/keys
Aug 08 03:59:40 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161580, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.251.212@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
Aug 08 03:59:41 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161580, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.251.212@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
Aug 08 03:59:43 dist-test-slave-pww0 krb5kdc[15269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 03:59:43.943042 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40281
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:40281
--ranger_config_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:35305/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 03:59:44.450872 16023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:59:44.451211 16023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:59:44.451342 16023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:59:44.459584 16023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 03:59:44.459722 16023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:59:44.459796 16023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:59:44.459869 16023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 03:59:44.459932 16023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 03:59:44.471447 16023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.251.212:35305/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:40281
--ranger_config_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:59:44.472922 16023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:59:44.474344 16023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:59:44.486565 16034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:59:44.486644 16037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:59:44.486683 16033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:59:44.487684 16023 server_base.cc:1034] running on GCE node
I20260808 03:59:44.488499 16023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:59:44.489982 16023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:59:44.491199 16023 hybrid_clock.cc:648] HybridClock initialized: now 1786161584491139 us; error 44 us; skew 500 ppm
I20260808 03:59:44.491605 16023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:59:44 dist-test-slave-pww0 krb5kdc[15269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161584, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:59:44.494874 16023 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 03:59:44.496694 16023 webserver.cc:460] Webserver started at http://127.28.251.254:45169/ using document root <none> and password file <none>
I20260808 03:59:44.497437 16023 fs_manager.cc:359] Metadata directory not provided
I20260808 03:59:44.497548 16023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:59:44.497829 16023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 03:59:44 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161584, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 03:59:44.551421 16023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "b1950583bec8410889ac0ae74c4bc850"
format_stamp: "Formatted at 2026-08-08 03:59:44 on dist-test-slave-pww0"
server_key: "8f354d1625082424ffb7f249db04dd38"
server_key_iv: "dfe14f5ac4ec8fa34f5e2d4882dabba4"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:44.552425 16023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "b1950583bec8410889ac0ae74c4bc850"
format_stamp: "Formatted at 2026-08-08 03:59:44 on dist-test-slave-pww0"
server_key: "8f354d1625082424ffb7f249db04dd38"
server_key_iv: "dfe14f5ac4ec8fa34f5e2d4882dabba4"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:44.558486 16023 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260808 03:59:44.576725 16048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:59:44.578332 16023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 03:59:44.578542 16023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "b1950583bec8410889ac0ae74c4bc850"
format_stamp: "Formatted at 2026-08-08 03:59:44 on dist-test-slave-pww0"
server_key: "8f354d1625082424ffb7f249db04dd38"
server_key_iv: "dfe14f5ac4ec8fa34f5e2d4882dabba4"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:44.578784 16023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:59:44.603956 16023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:59:44.608304 16023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:59:44.608626 16023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:59:44.628119 16023 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:40281
I20260808 03:59:44.628144 16142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:40281 every 8 connection(s)
I20260808 03:59:44.629664 16023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 03:59:44.633841 16143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:59:44.639108 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 16023
I20260808 03:59:44.639358 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 03:59:44.640240 29679 external_mini_cluster.cc:1428] Setting key a51f673c0f220e0ed59dd863f12ef712
I20260808 03:59:44.646559 16143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850: Bootstrap starting.
I20260808 03:59:44.650626 16143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850: Neither blocks nor log segments found. Creating new log.
I20260808 03:59:44.651975 16143 log.cc:824] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850: Log is configured to *not* fsync() on all Append() calls
I20260808 03:59:44.655745 16143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850: No bootstrap required, opened a new log
Aug 08 03:59:44 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:59:44.664150 16143 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1950583bec8410889ac0ae74c4bc850" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40281 } }
I20260808 03:59:44.664529 16143 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:59:44.664637 16143 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1950583bec8410889ac0ae74c4bc850, State: Initialized, Role: FOLLOWER
I20260808 03:59:44.664757 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:44.642715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.28.251.254:40281)
0808 03:59:44.643678 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:44.643691 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:44.643751 (+ 60us) server_negotiation.cc:407] Connection header received
0808 03:59:44.645566 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:44.645592 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:44.646033 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:44.646394 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:59:44.648105 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:44.649497 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:44.651333 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:44.652166 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:44.652320 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:44.657024 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:59:44.657033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:59:44.657049 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:59:44.657094 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:59:44.660295 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:44.661060 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:44.661068 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:44.661078 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:44.661317 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:44.661738 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:44.661743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:44.661746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:44.662382 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:59:44.662679 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:44.663199 (+ 520us) server_negotiation.cc:299] Negotiation successful
0808 03:59:44.663663 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":429,"threads_started":1}
I20260808 03:59:44.665294 16143 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1950583bec8410889ac0ae74c4bc850" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40281 } }
I20260808 03:59:44.665547 16143 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 03:59:44.665684 16143 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 03:59:44.665867 16143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:59:44.668995 16143 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1950583bec8410889ac0ae74c4bc850" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40281 } }
I20260808 03:59:44.669534 16143 leader_election.cc:302] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1950583bec8410889ac0ae74c4bc850; no voters:
I20260808 03:59:44.670113 16143 leader_election.cc:288] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 03:59:44.670259 16150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:59:44.670953 16150 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [term 1 LEADER]: Becoming Leader. State: Replica: b1950583bec8410889ac0ae74c4bc850, State: Running, Role: LEADER
I20260808 03:59:44.671563 16150 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1950583bec8410889ac0ae74c4bc850" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40281 } }
I20260808 03:59:44.672257 16143 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 03:59:44.674784 16153 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1950583bec8410889ac0ae74c4bc850. Latest consensus state: current_term: 1 leader_uuid: "b1950583bec8410889ac0ae74c4bc850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1950583bec8410889ac0ae74c4bc850" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40281 } } }
I20260808 03:59:44.675218 16153 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [sys.catalog]: This master's current role is: LEADER
I20260808 03:59:44.675665 16152 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1950583bec8410889ac0ae74c4bc850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1950583bec8410889ac0ae74c4bc850" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40281 } } }
I20260808 03:59:44.676095 16152 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850 [sys.catalog]: This master's current role is: LEADER
I20260808 03:59:44.677400 16165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 03:59:44.683194 16165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 03:59:44.683997 16143 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 03:59:44.705615 16165 catalog_manager.cc:1347] Generated new cluster ID: e2e5655692764a57b280a514891335b5
I20260808 03:59:44.705796 16165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 03:59:44.729393 16165 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 03:59:44.731650 16165 catalog_manager.cc:1504] Loading token signing keys...
I20260808 03:59:44.745424 16165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850: Generated new TSK 0
I20260808 03:59:44.746838 16165 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 03:59:45.563441 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:35305/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40281
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 03:59:45 dist-test-slave-pww0 krb5kdc[15269](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786161585, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 03:59:46.040964 16216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:59:46.041468 16216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:59:46.041697 16216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:59:46.052179 16216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:59:46.052335 16216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:59:46.052503 16216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 03:59:46.062984 16216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.251.212:35305/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:59:46.064608 16216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:59:46.066048 16216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:59:46.084125 16229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:59:46.086932 16231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:59:46.088649 16235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:59:46.097373 16216 server_base.cc:1034] running on GCE node
I20260808 03:59:46.098399 16216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:59:46.099915 16216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:59:46.101256 16216 hybrid_clock.cc:648] HybridClock initialized: now 1786161586101180 us; error 67 us; skew 500 ppm
I20260808 03:59:46.101723 16216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:59:46 dist-test-slave-pww0 krb5kdc[15269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161586, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:59:46.106513 16216 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 03:59:46.121320 16216 webserver.cc:460] Webserver started at http://127.28.251.193:39569/ using document root <none> and password file <none>
I20260808 03:59:46.122468 16216 fs_manager.cc:359] Metadata directory not provided
I20260808 03:59:46.122763 16216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:59:46.123318 16216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 03:59:46 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161586, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 03:59:46.147542 16216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "5e5fb140cacf4716a014030cfba3318f"
format_stamp: "Formatted at 2026-08-08 03:59:46 on dist-test-slave-pww0"
server_key: "654e565b1db4152e4040705ffb599c11"
server_key_iv: "34c93ee781792d80e4ba0db19394c158"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:46.149029 16216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "5e5fb140cacf4716a014030cfba3318f"
format_stamp: "Formatted at 2026-08-08 03:59:46 on dist-test-slave-pww0"
server_key: "654e565b1db4152e4040705ffb599c11"
server_key_iv: "34c93ee781792d80e4ba0db19394c158"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:46.170892 16216 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.009s sys 0.010s
I20260808 03:59:46.200631 16243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:59:46.203086 16216 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.002s
I20260808 03:59:46.203487 16216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "5e5fb140cacf4716a014030cfba3318f"
format_stamp: "Formatted at 2026-08-08 03:59:46 on dist-test-slave-pww0"
server_key: "654e565b1db4152e4040705ffb599c11"
server_key_iv: "34c93ee781792d80e4ba0db19394c158"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:46.203912 16216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:59:46.268188 16216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:59:46.277395 16216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:59:46.277763 16216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:59:46.288640 16216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Aug 08 03:59:46 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161585, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 03:59:46.304592 16216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:59:46.304818 16216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 03:59:46.305015 16216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:59:46.305105 16216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:59:46.455096 16216 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:41279
I20260808 03:59:46.455096 16438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:41279 every 8 connection(s)
I20260808 03:59:46.456664 16216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 03:59:46.463841 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 16216
I20260808 03:59:46.464080 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 03:59:46.465039 29679 external_mini_cluster.cc:1428] Setting key 4f647c71379e3f046a6a5a75d173b63b
Aug 08 03:59:46 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161586, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:59:46.484599 16441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:46.460196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:34697 (local address 127.28.251.254:40281)
0808 03:59:46.460725 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:46.460730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:46.461857 (+ 1127us) server_negotiation.cc:407] Connection header received
0808 03:59:46.464178 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:46.464185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:46.464329 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:46.464456 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:59:46.466740 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:46.467658 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:46.469352 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:46.469776 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:46.469894 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:46.473481 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:59:46.473486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:59:46.473490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:59:46.473527 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:59:46.476881 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:46.481014 (+ 4133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:46.481022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:46.481025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:46.481215 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:46.481966 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:46.481971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:46.481975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:46.482379 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:59:46.482522 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:46.483822 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0808 03:59:46.484179 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":275,"threads_started":1}
I20260808 03:59:46.487722 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:46.460860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40281 (local address 127.28.251.193:34697)
0808 03:59:46.461668 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0808 03:59:46.461713 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 03:59:46.463728 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:59:46.464954 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:59:46.464968 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:59:46.465640 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:59:46.466547 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:46.466572 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:46.467817 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:46.467822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:46.469187 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:46.469203 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:46.470032 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:46.470040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:46.470276 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:46.470978 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:59:46.471004 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:59:46.473234 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:59:46.479973 (+ 6739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:59:46.479986 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:59:46.480008 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:59:46.480825 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:59:46.481416 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:59:46.481421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:59:46.481425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:59:46.481815 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:59:46.482625 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:59:46.482634 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:59:46.483264 (+ 630us) client_negotiation.cc:769] Sending connection context
0808 03:59:46.485529 (+ 2265us) client_negotiation.cc:240] Negotiation successful
0808 03:59:46.486177 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":324,"threads_started":1}
I20260808 03:59:46.490301 16439 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40281
I20260808 03:59:46.490706 16439 heartbeater.cc:464] Registering TS with master...
I20260808 03:59:46.491842 16439 heartbeater.cc:511] Master 127.28.251.254:40281 requested a full tablet report, sending...
I20260808 03:59:46.494988 16078 ts_manager.cc:194] Registered new tserver with Master: 5e5fb140cacf4716a014030cfba3318f (127.28.251.193:41279)
I20260808 03:59:46.497279 16078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:34697
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 03:59:46.554611 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:35305/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40281
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 03:59:46 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161585, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
W20260808 03:59:47.104869 16447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:59:47.105222 16447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:59:47.105386 16447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:59:47.113260 16447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:59:47.113421 16447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:59:47.113588 16447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 03:59:47.124473 16447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.251.212:35305/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:59:47.126607 16447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:59:47.128664 16447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:59:47.144487 16472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:59:47.144663 16475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:59:47.144516 16473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:59:47.145551 16447 server_base.cc:1034] running on GCE node
I20260808 03:59:47.146286 16447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:59:47.147449 16447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:59:47.148713 16447 hybrid_clock.cc:648] HybridClock initialized: now 1786161587148645 us; error 59 us; skew 500 ppm
I20260808 03:59:47.149076 16447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:59:47 dist-test-slave-pww0 krb5kdc[15269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161587, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:59:47.152673 16447 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 03:59:47.154385 16447 webserver.cc:460] Webserver started at http://127.28.251.194:41121/ using document root <none> and password file <none>
I20260808 03:59:47.155139 16447 fs_manager.cc:359] Metadata directory not provided
I20260808 03:59:47.155272 16447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:59:47.155633 16447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 03:59:47 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161587, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 03:59:47.174510 16447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "7358dd03cb604a20b5b19e1d54b25201"
format_stamp: "Formatted at 2026-08-08 03:59:47 on dist-test-slave-pww0"
server_key: "d6b2f29ec31b86563cef858e5601bf24"
server_key_iv: "8e4e085eed00ee253f2ab80e5aa31fd8"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:47.175575 16447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "7358dd03cb604a20b5b19e1d54b25201"
format_stamp: "Formatted at 2026-08-08 03:59:47 on dist-test-slave-pww0"
server_key: "d6b2f29ec31b86563cef858e5601bf24"
server_key_iv: "8e4e085eed00ee253f2ab80e5aa31fd8"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:47.182219 16447 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 03:59:47.202283 16486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:59:47.204499 16447 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260808 03:59:47.204838 16447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "7358dd03cb604a20b5b19e1d54b25201"
format_stamp: "Formatted at 2026-08-08 03:59:47 on dist-test-slave-pww0"
server_key: "d6b2f29ec31b86563cef858e5601bf24"
server_key_iv: "8e4e085eed00ee253f2ab80e5aa31fd8"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:47.205260 16447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:59:47.239928 16447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:59:47.245451 16447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:59:47.245821 16447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:59:47.247133 16447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:59:47.249087 16447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:59:47.249186 16447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:59:47.249325 16447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:59:47.249384 16447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:59:47.297776 16447 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:36405
I20260808 03:59:47.297989 16684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:36405 every 8 connection(s)
I20260808 03:59:47.299501 16447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 03:59:47.301522 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 16447
I20260808 03:59:47.301931 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 03:59:47.302495 29679 external_mini_cluster.cc:1428] Setting key fc98d8b4e931ac7c16c5afa47c2b950e
Aug 08 03:59:47 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161587, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:59:47.344228 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:47.310025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:49621 (local address 127.28.251.254:40281)
0808 03:59:47.323037 (+ 13012us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:47.323042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:47.323061 (+ 19us) server_negotiation.cc:407] Connection header received
0808 03:59:47.323115 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:47.323119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:47.323188 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:47.323294 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:59:47.325630 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:47.326833 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:47.329521 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:47.330214 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:47.330383 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:47.336539 (+ 6156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:59:47.336548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:59:47.336553 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:59:47.336603 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:59:47.340125 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:47.341174 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:47.341180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:47.341184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:47.341355 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:47.341942 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:47.341947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:47.341951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:47.342348 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:59:47.342499 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:47.343557 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0808 03:59:47.343820 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12854,"thread_start_us":13442,"threads_started":1}
I20260808 03:59:47.345573 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:47.306212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40281 (local address 127.28.251.194:49621)
0808 03:59:47.320228 (+ 14016us) negotiation.cc:107] Waiting for socket to connect
0808 03:59:47.320296 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 03:59:47.322289 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:59:47.323739 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:59:47.323753 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:59:47.324459 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:59:47.325364 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:47.325394 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:47.327091 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:47.327097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:47.329256 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:47.329288 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:47.330533 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:47.330541 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:47.330876 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:47.333159 (+ 2283us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:59:47.333193 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:59:47.336170 (+ 2977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:59:47.340346 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:59:47.340357 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:59:47.340379 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:59:47.341007 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:59:47.341494 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:59:47.341499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:59:47.341502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:59:47.341811 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:59:47.342495 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:59:47.342503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:59:47.343040 (+ 537us) client_negotiation.cc:769] Sending connection context
0808 03:59:47.343371 (+ 331us) client_negotiation.cc:240] Negotiation successful
0808 03:59:47.343992 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13660,"thread_start_us":13360,"threads_started":1}
I20260808 03:59:47.347368 16686 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40281
I20260808 03:59:47.347805 16686 heartbeater.cc:464] Registering TS with master...
I20260808 03:59:47.348791 16686 heartbeater.cc:511] Master 127.28.251.254:40281 requested a full tablet report, sending...
I20260808 03:59:47.350792 16078 ts_manager.cc:194] Registered new tserver with Master: 7358dd03cb604a20b5b19e1d54b25201 (127.28.251.194:36405)
I20260808 03:59:47.351958 16078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:49621
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 03:59:47.394860 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:35305/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40281
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 03:59:47.502081 16439 heartbeater.cc:503] Master 127.28.251.254:40281 was elected leader, sending a full tablet report...
W20260808 03:59:47.765197 16695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 03:59:47.765542 16695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 03:59:47.765699 16695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 03:59:47.773419 16695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 03:59:47.773572 16695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 03:59:47.773739 16695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 03:59:47.784478 16695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.251.212:35305/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 03:59:47.785964 16695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 03:59:47.787360 16695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 03:59:47.801455 16711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:59:47.801481 16708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 03:59:47.801394 16715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 03:59:47.808216 16695 server_base.cc:1034] running on GCE node
I20260808 03:59:47.809042 16695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 03:59:47.810242 16695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 03:59:47.811478 16695 hybrid_clock.cc:648] HybridClock initialized: now 1786161587811406 us; error 44 us; skew 500 ppm
I20260808 03:59:47.811854 16695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 03:59:47 dist-test-slave-pww0 krb5kdc[15269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161587, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 03:59:47.815302 16695 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 03:59:47.816951 16695 webserver.cc:460] Webserver started at http://127.28.251.195:45387/ using document root <none> and password file <none>
I20260808 03:59:47.817708 16695 fs_manager.cc:359] Metadata directory not provided
I20260808 03:59:47.817842 16695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 03:59:47.818163 16695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 03:59:47 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161587, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 03:59:47.833491 16695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "5992734c96614aff957326cdcb43568e"
format_stamp: "Formatted at 2026-08-08 03:59:47 on dist-test-slave-pww0"
server_key: "a9a705a0beaef07b16534622874f618d"
server_key_iv: "33151d9a9377272198b08e9b55729aaa"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:47.834477 16695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "5992734c96614aff957326cdcb43568e"
format_stamp: "Formatted at 2026-08-08 03:59:47 on dist-test-slave-pww0"
server_key: "a9a705a0beaef07b16534622874f618d"
server_key_iv: "33151d9a9377272198b08e9b55729aaa"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:47.840610 16695 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 03:59:47.857579 16727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 03:59:47.859681 16695 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 03:59:47.859925 16695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "5992734c96614aff957326cdcb43568e"
format_stamp: "Formatted at 2026-08-08 03:59:47 on dist-test-slave-pww0"
server_key: "a9a705a0beaef07b16534622874f618d"
server_key_iv: "33151d9a9377272198b08e9b55729aaa"
server_key_version: "kuduclusterkey@0"
I20260808 03:59:47.860225 16695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 03:59:47.896657 16695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 03:59:47.900684 16695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 03:59:47.900990 16695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 03:59:47.902032 16695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 03:59:47.903590 16695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 03:59:47.903668 16695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 03:59:47.903770 16695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 03:59:47.903815 16695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 03:59:47.937180 16695 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:37585
I20260808 03:59:47.937228 16916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:37585 every 8 connection(s)
I20260808 03:59:47.938549 16695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 03:59:47.941033 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 16695
I20260808 03:59:47.941248 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 03:59:47.941797 29679 external_mini_cluster.cc:1428] Setting key 838d2f8a9484da513c796c08ad654ba7
Aug 08 03:59:47 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161587, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 03:59:47.960767 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:47.941404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:51737 (local address 127.28.251.254:40281)
0808 03:59:47.941914 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:47.941920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:47.943307 (+ 1387us) server_negotiation.cc:407] Connection header received
0808 03:59:47.945602 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:47.945607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:47.945718 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:47.945817 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:59:47.947439 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:47.948216 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:47.950000 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:47.950482 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:47.950603 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:47.954246 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:59:47.954252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:59:47.954256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:59:47.954295 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:59:47.957177 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:47.958192 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:47.958196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:47.958199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:47.958376 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:47.959002 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:47.959007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:47.959011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:47.959392 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:59:47.959573 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:47.960276 (+ 703us) server_negotiation.cc:299] Negotiation successful
0808 03:59:47.960490 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":237,"threads_started":1}
I20260808 03:59:47.961810 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:47.942193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40281 (local address 127.28.251.195:51737)
0808 03:59:47.943117 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0808 03:59:47.943172 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 03:59:47.945325 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:59:47.946101 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:59:47.946111 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:59:47.946587 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0808 03:59:47.947241 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:47.947261 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:47.948410 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:47.948415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:47.949822 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:47.949855 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:47.950732 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:47.950740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:47.950994 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:47.951715 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 03:59:47.951741 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 03:59:47.953955 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 03:59:47.957371 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:59:47.957383 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:59:47.957405 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:59:47.958046 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:59:47.958509 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 03:59:47.958515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 03:59:47.958518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 03:59:47.958869 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 03:59:47.959559 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 03:59:47.959566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 03:59:47.959942 (+ 376us) client_negotiation.cc:769] Sending connection context
0808 03:59:47.960202 (+ 260us) client_negotiation.cc:240] Negotiation successful
0808 03:59:47.960650 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":368,"threads_started":1}
I20260808 03:59:47.963301 16918 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40281
I20260808 03:59:47.963670 16918 heartbeater.cc:464] Registering TS with master...
I20260808 03:59:47.964478 16918 heartbeater.cc:511] Master 127.28.251.254:40281 requested a full tablet report, sending...
I20260808 03:59:47.966147 16078 ts_manager.cc:194] Registered new tserver with Master: 5992734c96614aff957326cdcb43568e (127.28.251.195:37585)
I20260808 03:59:47.967053 16078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:51737
I20260808 03:59:47.970561 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 03:59:47.990367 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:47.975537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.28.251.254:40281)
0808 03:59:47.975726 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:47.975733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:47.976246 (+ 513us) server_negotiation.cc:407] Connection header received
0808 03:59:47.976446 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:47.976450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:47.976588 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:47.976681 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:59:47.978425 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:47.979174 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:47.980705 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:47.981174 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:47.981302 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:47.984044 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:59:47.984050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:59:47.984053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:59:47.984100 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:59:47.987331 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:47.988270 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:47.988275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:47.988278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:47.988426 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:47.988842 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:47.988846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:47.988848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:47.989156 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:59:47.989401 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:47.989788 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 03:59:47.990044 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 03:59:47.995048 16078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 03:59:48.003147 16078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 03:59:48.045915 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.018266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.28.251.194:36405)
0808 03:59:48.031239 (+ 12973us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:48.031251 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:48.036105 (+ 4854us) server_negotiation.cc:407] Connection header received
0808 03:59:48.036492 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:48.036501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:48.036760 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:48.036961 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:59:48.038017 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.039560 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.043228 (+ 3668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.044516 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.044633 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.044765 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:48.045333 (+ 568us) server_negotiation.cc:299] Negotiation successful
0808 03:59:48.045522 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12763,"thread_start_us":12464,"threads_started":1}
I20260808 03:59:48.046373 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.018651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:36405 (local address 127.0.0.1:53958)
0808 03:59:48.035954 (+ 17303us) negotiation.cc:107] Waiting for socket to connect
0808 03:59:48.035976 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 03:59:48.036246 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:59:48.037353 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:59:48.037358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:59:48.037384 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:59:48.037810 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.037823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.039730 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.039735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.042997 (+ 3262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.043017 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.044927 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.044932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.045084 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.045124 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 03:59:48.045248 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 03:59:48.045491 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17187,"mutex_wait_us":619,"thread_start_us":16041,"threads_started":1}
I20260808 03:59:48.046819 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.018252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:41279 (local address 127.0.0.1:60236)
0808 03:59:48.035173 (+ 16921us) negotiation.cc:107] Waiting for socket to connect
0808 03:59:48.035226 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 03:59:48.035504 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:59:48.036785 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:59:48.036794 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:59:48.036829 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:59:48.037377 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.037394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.039656 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.039663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.043399 (+ 3736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.043409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.045821 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.045828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.046038 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.046101 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 03:59:48.046204 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 03:59:48.046361 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16638,"mutex_wait_us":128,"thread_start_us":17423,"threads_started":1}
I20260808 03:59:48.047375 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.018364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.28.251.193:41279)
0808 03:59:48.034241 (+ 15877us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:48.034252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:48.035449 (+ 1197us) server_negotiation.cc:407] Connection header received
0808 03:59:48.035803 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:48.035811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:48.036164 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:48.036413 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:59:48.037634 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.039420 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.043659 (+ 4239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.045188 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.045411 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.045606 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:48.046504 (+ 898us) server_negotiation.cc:299] Negotiation successful
0808 03:59:48.046691 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15638,"thread_start_us":15897,"threads_started":1}
I20260808 03:59:48.050197 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.037789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:37585 (local address 127.0.0.1:40714)
0808 03:59:48.042008 (+ 4219us) negotiation.cc:107] Waiting for socket to connect
0808 03:59:48.042025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 03:59:48.042196 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:59:48.042816 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:59:48.042821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:59:48.042850 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:59:48.043376 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.043390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.045454 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.045459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.047725 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.047742 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.049374 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.049381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.049555 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.049589 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 03:59:48.049676 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 03:59:48.049827 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4093,"thread_start_us":3870,"threads_started":1}
I20260808 03:59:48.050282 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.036650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.28.251.195:37585)
0808 03:59:48.041632 (+ 4982us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:48.041641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:48.042149 (+ 508us) server_negotiation.cc:407] Connection header received
0808 03:59:48.042370 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:48.042377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:48.042609 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:48.042807 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:59:48.043672 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.045262 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.047937 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.049141 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.049368 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.049570 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:48.049757 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 03:59:48.049937 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4770,"thread_start_us":4549,"threads_started":1}
I20260808 03:59:48.051182 16325 tablet_service.cc:1467] Processing CreateTablet for tablet 97fa678e93884df0ab8017bb286bcfbf (DEFAULT_TABLE table=test-table [id=1ae13ec4d2c04a35a236d4cc339ffda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:59:48.051182 16578 tablet_service.cc:1467] Processing CreateTablet for tablet 97fa678e93884df0ab8017bb286bcfbf (DEFAULT_TABLE table=test-table [id=1ae13ec4d2c04a35a236d4cc339ffda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:59:48.052896 16578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97fa678e93884df0ab8017bb286bcfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:59:48.052898 16325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97fa678e93884df0ab8017bb286bcfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:59:48.054111 16807 tablet_service.cc:1467] Processing CreateTablet for tablet 97fa678e93884df0ab8017bb286bcfbf (DEFAULT_TABLE table=test-table [id=1ae13ec4d2c04a35a236d4cc339ffda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 03:59:48.055452 16807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97fa678e93884df0ab8017bb286bcfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 03:59:48.067936 16951 tablet_bootstrap.cc:492] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e: Bootstrap starting.
I20260808 03:59:48.071768 16951 tablet_bootstrap.cc:654] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e: Neither blocks nor log segments found. Creating new log.
I20260808 03:59:48.072356 16953 tablet_bootstrap.cc:492] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f: Bootstrap starting.
I20260808 03:59:48.074429 16955 tablet_bootstrap.cc:492] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201: Bootstrap starting.
I20260808 03:59:48.075395 16951 log.cc:824] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e: Log is configured to *not* fsync() on all Append() calls
I20260808 03:59:48.076470 16953 tablet_bootstrap.cc:654] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f: Neither blocks nor log segments found. Creating new log.
I20260808 03:59:48.077739 16953 log.cc:824] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f: Log is configured to *not* fsync() on all Append() calls
I20260808 03:59:48.078050 16955 tablet_bootstrap.cc:654] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201: Neither blocks nor log segments found. Creating new log.
I20260808 03:59:48.079099 16951 tablet_bootstrap.cc:492] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e: No bootstrap required, opened a new log
I20260808 03:59:48.079311 16955 log.cc:824] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201: Log is configured to *not* fsync() on all Append() calls
I20260808 03:59:48.079434 16951 ts_tablet_manager.cc:1397] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20260808 03:59:48.081205 16953 tablet_bootstrap.cc:492] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f: No bootstrap required, opened a new log
I20260808 03:59:48.081491 16953 ts_tablet_manager.cc:1397] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260808 03:59:48.081951 16955 tablet_bootstrap.cc:492] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201: No bootstrap required, opened a new log
I20260808 03:59:48.082190 16955 ts_tablet_manager.cc:1397] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 03:59:48.090375 16955 raft_consensus.cc:348] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5fb140cacf4716a014030cfba3318f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 41279 } } peers { permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 } } peers { permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 } }
I20260808 03:59:48.090798 16955 raft_consensus.cc:374] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:59:48.090900 16955 raft_consensus.cc:729] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7358dd03cb604a20b5b19e1d54b25201, State: Initialized, Role: FOLLOWER
I20260808 03:59:48.091413 16951 raft_consensus.cc:348] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5fb140cacf4716a014030cfba3318f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 41279 } } peers { permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 } } peers { permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 } }
I20260808 03:59:48.091650 16955 consensus_queue.cc:260] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5fb140cacf4716a014030cfba3318f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 41279 } } peers { permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 } } peers { permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 } }
I20260808 03:59:48.091967 16951 raft_consensus.cc:374] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:59:48.092104 16951 raft_consensus.cc:729] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5992734c96614aff957326cdcb43568e, State: Initialized, Role: FOLLOWER
I20260808 03:59:48.092139 16953 raft_consensus.cc:348] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5fb140cacf4716a014030cfba3318f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 41279 } } peers { permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 } } peers { permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 } }
I20260808 03:59:48.092583 16953 raft_consensus.cc:374] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 03:59:48.092702 16953 raft_consensus.cc:729] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5fb140cacf4716a014030cfba3318f, State: Initialized, Role: FOLLOWER
I20260808 03:59:48.092881 16951 consensus_queue.cc:260] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5fb140cacf4716a014030cfba3318f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 41279 } } peers { permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 } } peers { permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 } }
I20260808 03:59:48.093152 16686 heartbeater.cc:503] Master 127.28.251.254:40281 was elected leader, sending a full tablet report...
I20260808 03:59:48.093389 16953 consensus_queue.cc:260] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5fb140cacf4716a014030cfba3318f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 41279 } } peers { permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 } } peers { permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 } }
I20260808 03:59:48.093860 16955 ts_tablet_manager.cc:1428] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260808 03:59:48.095021 16918 heartbeater.cc:503] Master 127.28.251.254:40281 was elected leader, sending a full tablet report...
I20260808 03:59:48.097633 16951 ts_tablet_manager.cc:1428] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e: Time spent starting tablet: real 0.018s user 0.012s sys 0.003s
I20260808 03:59:48.098953 16953 ts_tablet_manager.cc:1428] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260808 03:59:48.100788 16966 raft_consensus.cc:482] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:59:48.101101 16966 raft_consensus.cc:504] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5fb140cacf4716a014030cfba3318f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 41279 } } peers { permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 } } peers { permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 } }
I20260808 03:59:48.104101 16966 leader_election.cc:288] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7358dd03cb604a20b5b19e1d54b25201 (127.28.251.194:36405), 5992734c96614aff957326cdcb43568e (127.28.251.195:37585)
I20260808 03:59:48.109721 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.104012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:54531 (local address 127.28.251.195:37585)
0808 03:59:48.104215 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:48.104221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:48.104295 (+ 74us) server_negotiation.cc:407] Connection header received
0808 03:59:48.104555 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:48.104561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:48.104625 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:48.104701 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:59:48.105398 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.106188 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.107983 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.108806 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.108962 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.109043 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:48.109339 (+ 296us) server_negotiation.cc:299] Negotiation successful
0808 03:59:48.109474 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 03:59:48.109856 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.103528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:37585 (local address 127.28.251.193:54531)
0808 03:59:48.104182 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0808 03:59:48.104203 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 03:59:48.104412 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:59:48.104741 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:59:48.104746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:59:48.104773 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:59:48.105247 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.105259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.106322 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.106328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.107835 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.107848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.109013 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.109019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.109180 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.109206 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 03:59:48.109291 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 03:59:48.109431 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":323,"threads_started":1}
I20260808 03:59:48.110657 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.103518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:59779 (local address 127.28.251.194:36405)
0808 03:59:48.103840 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:48.103847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:48.105410 (+ 1563us) server_negotiation.cc:407] Connection header received
0808 03:59:48.105575 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:48.105581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:48.105660 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:48.105742 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:59:48.106294 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.107031 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.108755 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.109549 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.109694 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.109769 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:48.110245 (+ 476us) server_negotiation.cc:299] Negotiation successful
0808 03:59:48.110420 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260808 03:59:48.110739 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.103538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:36405 (local address 127.28.251.193:59779)
0808 03:59:48.105294 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0808 03:59:48.105310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 03:59:48.105446 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:59:48.105762 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:59:48.105767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:59:48.105792 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:59:48.106129 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.106139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.107171 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.107177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.108595 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.108609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.109776 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.109783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.110076 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.110112 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 03:59:48.110195 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 03:59:48.110326 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1622,"mutex_wait_us":22,"thread_start_us":1356,"threads_started":1}
I20260808 03:59:48.110939 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fa678e93884df0ab8017bb286bcfbf" candidate_uuid: "5e5fb140cacf4716a014030cfba3318f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5992734c96614aff957326cdcb43568e" is_pre_election: true
I20260808 03:59:48.111651 16963 raft_consensus.cc:482] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 03:59:48.111685 16844 raft_consensus.cc:2454] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5fb140cacf4716a014030cfba3318f in term 0.
I20260808 03:59:48.111910 16614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fa678e93884df0ab8017bb286bcfbf" candidate_uuid: "5e5fb140cacf4716a014030cfba3318f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358dd03cb604a20b5b19e1d54b25201" is_pre_election: true
I20260808 03:59:48.111938 16963 raft_consensus.cc:504] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5fb140cacf4716a014030cfba3318f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 41279 } } peers { permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 } } peers { permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 } }
I20260808 03:59:48.112486 16248 leader_election.cc:302] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5992734c96614aff957326cdcb43568e, 5e5fb140cacf4716a014030cfba3318f; no voters:
I20260808 03:59:48.112949 16966 raft_consensus.cc:2781] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 03:59:48.113021 16614 raft_consensus.cc:2454] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5fb140cacf4716a014030cfba3318f in term 0.
I20260808 03:59:48.113126 16966 raft_consensus.cc:482] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 03:59:48.113230 16966 raft_consensus.cc:3037] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:59:48.114075 16963 leader_election.cc:288] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e5fb140cacf4716a014030cfba3318f (127.28.251.193:41279), 5992734c96614aff957326cdcb43568e (127.28.251.195:37585)
I20260808 03:59:48.116670 16966 raft_consensus.cc:504] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5fb140cacf4716a014030cfba3318f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 41279 } } peers { permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 } } peers { permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 } }
I20260808 03:59:48.118019 16614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fa678e93884df0ab8017bb286bcfbf" candidate_uuid: "5e5fb140cacf4716a014030cfba3318f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358dd03cb604a20b5b19e1d54b25201"
I20260808 03:59:48.118072 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fa678e93884df0ab8017bb286bcfbf" candidate_uuid: "5e5fb140cacf4716a014030cfba3318f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5992734c96614aff957326cdcb43568e"
I20260808 03:59:48.118274 16614 raft_consensus.cc:3037] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:59:48.118319 16844 raft_consensus.cc:3037] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e [term 0 FOLLOWER]: Advancing to term 1
I20260808 03:59:48.121780 16844 raft_consensus.cc:2454] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5fb140cacf4716a014030cfba3318f in term 1.
I20260808 03:59:48.122009 16614 raft_consensus.cc:2454] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5fb140cacf4716a014030cfba3318f in term 1.
I20260808 03:59:48.122299 16248 leader_election.cc:302] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5992734c96614aff957326cdcb43568e, 5e5fb140cacf4716a014030cfba3318f; no voters:
I20260808 03:59:48.122926 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.114383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:37155 (local address 127.28.251.193:41279)
0808 03:59:48.114601 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:48.114607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:48.115032 (+ 425us) server_negotiation.cc:407] Connection header received
0808 03:59:48.115718 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:48.115724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:48.115789 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:48.115876 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:59:48.116740 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.117715 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.120488 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.121706 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.121852 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.121933 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:48.122572 (+ 639us) server_negotiation.cc:299] Negotiation successful
0808 03:59:48.122710 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 03:59:48.123746 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.115278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:42259 (local address 127.28.251.195:37585)
0808 03:59:48.115491 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:48.115497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:48.115859 (+ 362us) server_negotiation.cc:407] Connection header received
0808 03:59:48.116005 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:48.116009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:48.116083 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:48.116180 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 03:59:48.116997 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.118059 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.120765 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.122003 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.122130 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.122202 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:48.123251 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0808 03:59:48.123419 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 03:59:48.124598 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.115075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:37585 (local address 127.28.251.194:42259)
0808 03:59:48.115746 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0808 03:59:48.115764 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 03:59:48.115897 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:59:48.116209 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:59:48.116213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:59:48.116266 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:59:48.116803 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.116817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.118248 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.118253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.120547 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.120567 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.122795 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.122807 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.123032 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.123071 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 03:59:48.123955 (+ 884us) client_negotiation.cc:240] Negotiation successful
0808 03:59:48.124130 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":24,"spinlock_wait_cycles":1408,"thread_start_us":294,"threads_started":1}
I20260808 03:59:48.124755 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.114276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:41279 (local address 127.28.251.194:37155)
0808 03:59:48.114936 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0808 03:59:48.114956 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 03:59:48.115513 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 03:59:48.116040 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 03:59:48.116045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 03:59:48.116072 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 03:59:48.116593 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.116604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.117860 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.117865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.120280 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 03:59:48.120295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.122166 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.122173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 03:59:48.122383 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.122418 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 03:59:48.123122 (+ 704us) client_negotiation.cc:240] Negotiation successful
0808 03:59:48.123826 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":35,"spinlock_wait_cycles":28800,"thread_start_us":333,"threads_started":1}
I20260808 03:59:48.125607 16844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fa678e93884df0ab8017bb286bcfbf" candidate_uuid: "7358dd03cb604a20b5b19e1d54b25201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5992734c96614aff957326cdcb43568e" is_pre_election: true
I20260808 03:59:48.125977 16844 raft_consensus.cc:2379] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7358dd03cb604a20b5b19e1d54b25201 in current term 1: Already voted for candidate 5e5fb140cacf4716a014030cfba3318f in this term.
I20260808 03:59:48.125964 16360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fa678e93884df0ab8017bb286bcfbf" candidate_uuid: "7358dd03cb604a20b5b19e1d54b25201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5fb140cacf4716a014030cfba3318f" is_pre_election: true
I20260808 03:59:48.126511 16360 raft_consensus.cc:2379] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7358dd03cb604a20b5b19e1d54b25201 in current term 1: Already voted for candidate 5e5fb140cacf4716a014030cfba3318f in this term.
I20260808 03:59:48.126540 16966 leader_election.cc:288] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [CANDIDATE]: Term 1 election: Requested vote from peers 7358dd03cb604a20b5b19e1d54b25201 (127.28.251.194:36405), 5992734c96614aff957326cdcb43568e (127.28.251.195:37585)
I20260808 03:59:48.126782 16966 raft_consensus.cc:2781] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 03:59:48.126940 16966 raft_consensus.cc:686] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 1 LEADER]: Becoming Leader. State: Replica: 5e5fb140cacf4716a014030cfba3318f, State: Running, Role: LEADER
I20260808 03:59:48.127426 16966 consensus_queue.cc:237] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5fb140cacf4716a014030cfba3318f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 41279 } } peers { permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 } } peers { permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 } }
I20260808 03:59:48.127504 16492 leader_election.cc:302] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7358dd03cb604a20b5b19e1d54b25201; no voters: 5992734c96614aff957326cdcb43568e, 5e5fb140cacf4716a014030cfba3318f
I20260808 03:59:48.127913 16963 raft_consensus.cc:2726] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 03:59:48.134821 16078 catalog_manager.cc:5495] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e5fb140cacf4716a014030cfba3318f (127.28.251.193). New cstate: current_term: 1 leader_uuid: "5e5fb140cacf4716a014030cfba3318f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5fb140cacf4716a014030cfba3318f" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 41279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260808 03:59:48.164206 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.158194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.28.251.193:41279)
0808 03:59:48.158410 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:48.158417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:48.158439 (+ 22us) server_negotiation.cc:407] Connection header received
0808 03:59:48.158690 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:48.158695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:48.158790 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:48.158938 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 03:59:48.159356 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.160165 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.161672 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.162120 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.162241 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.162925 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 03:59:48.163545 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 03:59:48.163755 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:48.163811 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 03:59:48.163906 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 03:59:48.173378 16844 raft_consensus.cc:1261] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e [term 1 FOLLOWER]: Refusing update from remote peer 5e5fb140cacf4716a014030cfba3318f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:59:48.173461 16614 raft_consensus.cc:1261] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 1 FOLLOWER]: Refusing update from remote peer 5e5fb140cacf4716a014030cfba3318f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 03:59:48.174485 16974 consensus_queue.cc:1035] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7358dd03cb604a20b5b19e1d54b25201" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 36405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:59:48.174871 16966 consensus_queue.cc:1035] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5992734c96614aff957326cdcb43568e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 03:59:48.211004 16077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:59:48.211481 16077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 03:59:48.216615 16077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1950583bec8410889ac0ae74c4bc850: Sending DeleteTablet for 3 replicas of tablet 97fa678e93884df0ab8017bb286bcfbf
I20260808 03:59:48.218058 16578 tablet_service.cc:1514] Processing DeleteTablet for tablet 97fa678e93884df0ab8017bb286bcfbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:59:48 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:53958
I20260808 03:59:48.218058 16807 tablet_service.cc:1514] Processing DeleteTablet for tablet 97fa678e93884df0ab8017bb286bcfbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:59:48 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:40714
I20260808 03:59:48.218907 16325 tablet_service.cc:1514] Processing DeleteTablet for tablet 97fa678e93884df0ab8017bb286bcfbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 03:59:48 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:60236
I20260808 03:59:48.219244 16997 tablet_replica.cc:330] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201: stopping tablet replica
I20260808 03:59:48.220320 16997 raft_consensus.cc:2229] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:59:48.221026 16997 raft_consensus.cc:2258] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:59:48.221752 16999 tablet_replica.cc:330] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e: stopping tablet replica
I20260808 03:59:48.222026 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 16216
I20260808 03:59:48.222692 16999 raft_consensus.cc:2229] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 03:59:48.223291 16999 raft_consensus.cc:2258] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:59:48.229211 16999 ts_tablet_manager.cc:1905] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:59:48.229338 16997 ts_tablet_manager.cc:1905] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 03:59:48.238401 16997 ts_tablet_manager.cc:1918] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:59:48.238657 16997 log.cc:1196] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/97fa678e93884df0ab8017bb286bcfbf
I20260808 03:59:48.238842 17000 tablet_replica.cc:330] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f: stopping tablet replica
I20260808 03:59:48.239231 16997 ts_tablet_manager.cc:1939] T 97fa678e93884df0ab8017bb286bcfbf P 7358dd03cb604a20b5b19e1d54b25201: Deleting consensus metadata
I20260808 03:59:48.241490 16999 ts_tablet_manager.cc:1918] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:59:48.241480 16053 catalog_manager.cc:4841] TS 7358dd03cb604a20b5b19e1d54b25201 (127.28.251.194:36405): tablet 97fa678e93884df0ab8017bb286bcfbf (table test-table [id=1ae13ec4d2c04a35a236d4cc339ffda2]) successfully deleted
I20260808 03:59:48.241703 16999 log.cc:1196] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/97fa678e93884df0ab8017bb286bcfbf
I20260808 03:59:48.241930 17000 raft_consensus.cc:2229] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 1 LEADER]: Raft consensus shutting down.
I20260808 03:59:48.242242 16999 ts_tablet_manager.cc:1939] T 97fa678e93884df0ab8017bb286bcfbf P 5992734c96614aff957326cdcb43568e: Deleting consensus metadata
I20260808 03:59:48.242740 17000 raft_consensus.cc:2258] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 03:59:48.244330 16055 catalog_manager.cc:4841] TS 5992734c96614aff957326cdcb43568e (127.28.251.195:37585): tablet 97fa678e93884df0ab8017bb286bcfbf (table test-table [id=1ae13ec4d2c04a35a236d4cc339ffda2]) successfully deleted
I20260808 03:59:48.246201 17000 ts_tablet_manager.cc:1905] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 03:59:48 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 03:59:48.253887 17000 ts_tablet_manager.cc:1918] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 03:59:48.254123 17000 log.cc:1196] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/97fa678e93884df0ab8017bb286bcfbf
I20260808 03:59:48.254770 17000 ts_tablet_manager.cc:1939] T 97fa678e93884df0ab8017bb286bcfbf P 5e5fb140cacf4716a014030cfba3318f: Deleting consensus metadata
I20260808 03:59:48.257108 16055 catalog_manager.cc:4841] TS 5e5fb140cacf4716a014030cfba3318f (127.28.251.193:41279): tablet 97fa678e93884df0ab8017bb286bcfbf (table test-table [id=1ae13ec4d2c04a35a236d4cc339ffda2]) successfully deleted
I20260808 03:59:48.258026 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.224524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.28.251.193:41279)
0808 03:59:48.224769 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:48.224776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:48.224810 (+ 34us) server_negotiation.cc:407] Connection header received
0808 03:59:48.224887 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:48.224893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:48.224991 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:48.225140 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:59:48.234418 (+ 9278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.235817 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.239529 (+ 3712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.240264 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.240388 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.248198 (+ 7810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:59:48.248217 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:59:48.248238 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:59:48.248289 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:59:48.252951 (+ 4662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:48.254154 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:48.254166 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:48.254170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:48.254390 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:48.255009 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:48.255018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:48.255021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:48.255699 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:59:48.256044 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:48.256780 (+ 736us) server_negotiation.cc:299] Negotiation successful
0808 03:59:48.257668 (+ 888us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 03:59:48.259769 16432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:59:48.379832 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 16216
I20260808 03:59:48.410667 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 16447
Aug 08 03:59:48 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 03:59:48.427181 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.411604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.28.251.194:36405)
0808 03:59:48.411791 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:48.411796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:48.411871 (+ 75us) server_negotiation.cc:407] Connection header received
0808 03:59:48.412022 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:48.412026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:48.412102 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:48.412220 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:59:48.413585 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.414390 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.415809 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.416265 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.416395 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.420860 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:59:48.420879 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:59:48.420896 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:59:48.420940 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:59:48.424331 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:48.425166 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:48.425177 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:48.425180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:48.425349 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:48.425723 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:48.425728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:48.425730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:48.426185 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:59:48.426422 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:48.426720 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 03:59:48.426908 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 03:59:48.428124 16678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:59:48.521548 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 16447
I20260808 03:59:48.550274 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 16695
Aug 08 03:59:48 dist-test-slave-pww0 krb5kdc[15269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 03:59:48.566710 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 03:59:48.551186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.28.251.195:37585)
0808 03:59:48.551363 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 03:59:48.551368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 03:59:48.551389 (+ 21us) server_negotiation.cc:407] Connection header received
0808 03:59:48.551624 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 03:59:48.551629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 03:59:48.551718 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 03:59:48.551809 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 03:59:48.553187 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.553876 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.555224 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 03:59:48.555741 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 03:59:48.555874 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 03:59:48.560486 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 03:59:48.560504 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 03:59:48.560521 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 03:59:48.560573 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 03:59:48.563702 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:48.564486 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:48.564495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:48.564497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:48.564642 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 03:59:48.565049 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 03:59:48.565052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 03:59:48.565054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 03:59:48.565462 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 03:59:48.565796 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 03:59:48.566146 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 03:59:48.566380 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 03:59:48.567442 16911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:59:48.660344 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 16695
I20260808 03:59:48.683099 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 16023
I20260808 03:59:48.684326 16135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 03:59:48.800798 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 16023
I20260808 03:59:48.832015 29679 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 03:59:49.178166 29679 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 03:59:49.178463 29679 mini_ranger.cc:66] Stopping Ranger...
I20260808 03:59:49.525189 29679 mini_ranger.cc:68] Stopped Ranger
2026-08-08 03:59:49.525 UTC [15317] LOG: received smart shutdown request
2026-08-08 03:59:49.529 UTC [15317] LOG: background worker "logical replication launcher" (PID 15335) exited with exit code 1
2026-08-08 03:59:49.529 UTC [15321] LOG: shutting down
2026-08-08 03:59:49.697 UTC [15317] LOG: database system is shut down
2026-08-08T03:59:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53158 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 03:59:49 dist-test-slave-pww0 krb5kdc[17024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 03:59:49 dist-test-slave-pww0 krb5kdc[17024](info): set up 2 sockets
Aug 08 03:59:49 dist-test-slave-pww0 krb5kdc[17024](info): commencing operation
krb5kdc: starting...
W20260808 03:59:51.089999 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.001s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 03:59:51 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T03:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T03:59:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/rangeradmin_127.28.251.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/rangeradmin_127.28.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/rangerlookup_127.28.251.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/rangerlookup_127.28.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/HTTP_127.28.251.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/HTTP_127.28.251.212@KRBTEST.COM.keytab.
I20260808 03:59:51.552026 29679 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhSn85p/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/postgres -l logfile start
2026-08-08 03:59:54.051 UTC [17078] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 03:59:54.051 UTC [17078] LOG: listening on IPv4 address "127.28.251.212", port 53401
2026-08-08 03:59:54.056 UTC [17078] LOG: listening on Unix socket "/tmp/.s.PGSQL.53401"
2026-08-08 03:59:54.081 UTC [17082] LOG: database system was shut down at 2026-08-08 03:59:52 UTC
2026-08-08 03:59:54.084 UTC [17078] LOG: database system is ready to accept connections
I20260808 03:59:55.210117 29679 mini_postgres.cc:96] Postgres bound to 53401
2026-08-08 03:59:55.237 UTC [17095] FATAL: database "slave" does not exist
127.28.251.212:53401 - accepting connections
I20260808 03:59:55.238404 29679 mini_ranger.cc:161] Starting Ranger...
I20260808 03:59:55.275108 29679 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 03:59:55.758086 29679 mini_ranger.cc:87] Created ranger Postgres database
I20260808 03:59:55.758222 29679 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-admin
2026-08-08 03:59:56,064 [I] DB FLAVOR :POSTGRES
2026-08-08 03:59:56,065 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 03:59:56,065 [I] Checking connection..
2026-08-08 03:59:56,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 03:59:56,346 [I] Checking connection passed.
2026-08-08 03:59:56,346 [I] --------- Verifying version history table ---------
2026-08-08 03:59:56,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 03:59:56,624 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 03:59:56,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 03:59:56,899 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 03:59:56,899 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 03:59:56,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 03:59:57.140 UTC [17203] WARNING: there is no transaction in progress
2026-08-08 03:59:57,149 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 03:59:57,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 03:59:57,412 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 03:59:57,413 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 03:59:57,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 03:59:57,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 03:59:57,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pww0.c.gcp-upstream.internal','N') ;"
2026-08-08 03:59:58,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 03:59:58,449 [I] Table x_portal_user does not exist in database ranger
2026-08-08 03:59:58,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 03:59:58,706 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 03:59:58,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 03:59:58,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 03:59:59,244 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 03:59:59,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 04:00:00.239 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.256 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.275 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.290 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.306 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.325 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.389 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.397 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.411 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.422 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.433 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.448 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.458 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.465 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.473 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:00.480 UTC [17386] WARNING: there is no transaction in progress
2026-08-08 04:00:01,196 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 04:00:01,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 04:00:01,466 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 04:00:01,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:00:01,743 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 04:00:01,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:00:02,011 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 04:00:02,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:00:02,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:00:02,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pww0.c.gcp-upstream.internal';"
2026-08-08 04:00:02,822 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 04:00:02,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 04:00:03,097 [I] DB_PATCHES have already been applied
I20260808 04:00:03.103919 29679 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhSn85p/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 04:00:03.104015 29679 mini_ranger.cc:193] Using host: 127.28.251.212
I20260808 04:00:03.131681 29679 mini_ranger.cc:227] Ranger admin URL: http://127.28.251.212:52309
Aug 08, 2026 4:00:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:00:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:00:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 4:00:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 4:00:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 4:00:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 4:00:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 04:00:04.425900 29679 mini_ranger.cc:160] Time spent starting Ranger: real 9.187s user 0.000s sys 0.083s
Aug 08, 2026 4:00:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 4:00:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 4:00:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 4:00:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 4:00:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 04:00:28.113979 29679 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/rangerkms_127.28.251.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/rangerkms_127.28.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.251.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.251.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.251.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/HTTP_127.28.251.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.251.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/HTTP_127.28.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 04:00:28 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161628, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 04:00:28.353711 29679 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 04:00:28.387388 29679 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 04:00:29.266944 29679 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 04:00:29.267079 29679 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-kms
I20260808 04:00:29.267122 29679 mini_ranger_kms.cc:227] Using postgres at 127.28.251.212:53401
2026-08-08 04:00:29,697 [I] DB FLAVOR :POSTGRES
2026-08-08 04:00:29,697 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:00:29,697 [I] Checking connection
2026-08-08 04:00:29,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 04:00:30,034 [I] connection success
2026-08-08 04:00:30,035 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 04:00:30,035 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 04:00:30,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 04:00:30,318 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 04:00:30,319 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:00:30,319 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 04:00:30,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.251.212:53401/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 04:00:30,599 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 04:00:31.128098 29679 mini_ranger_kms.cc:310] Created kms service
I20260808 04:00:31.323670 29679 mini_ranger_kms.cc:326] Created kudu user
I20260808 04:00:31.373302 29679 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 04:00:31 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161628, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 04:00:31.809566 29679 mini_ranger_kms.cc:384] Added ranger policy
I20260808 04:00:31.809917 29679 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhSn85p/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhSn85p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 04:00:31.809970 29679 mini_ranger_kms.cc:240] Using host: 127.28.251.212
I20260808 04:00:31.835947 29679 mini_ranger_kms.cc:276] Ranger KMS PID: 17705
I20260808 04:00:31.836091 29679 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.251.212:58079
Aug 08, 2026 4:00:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:00:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:00:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 4:00:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 4:00:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 4:00:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 04:00:34 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786161634, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.251.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:34.503865 29679 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.150s user 0.106s sys 0.364s
I20260808 04:00:34.504101 29679 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 04:00:34.504267 29679 mini_ranger_kms.cc:401] 127.28.251.212:58079/kms/v1/keys
Aug 08 04:00:34 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161634, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.251.212@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
Aug 08 04:00:35 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161634, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.251.212@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
Aug 08 04:00:37 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:00:37.818605 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:38543
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:38543
--ranger_config_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:58079/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:00:38.192938 17784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:38.193300 17784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:38.193451 17784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:38.201335 17784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:00:38.201483 17784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:38.201570 17784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:38.201668 17784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:00:38.201747 17784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:00:38.213231 17784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.251.212:58079/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:38543
--ranger_config_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:38543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:38.214785 17784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:38.216305 17784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:38.228957 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:38.228997 17794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:38.229019 17797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:38.229362 17784 server_base.cc:1034] running on GCE node
I20260808 04:00:38.230449 17784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:38.232156 17784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:38.233389 17784 hybrid_clock.cc:648] HybridClock initialized: now 1786161638233343 us; error 41 us; skew 500 ppm
I20260808 04:00:38.233783 17784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:38 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161638, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:38.241451 17784 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:00:38.243243 17784 webserver.cc:460] Webserver started at http://127.28.251.254:43861/ using document root <none> and password file <none>
I20260808 04:00:38.244020 17784 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:38.244138 17784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:38.244436 17784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:00:38 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161638, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 04:00:38.298808 17784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "9bb7457951c040a2a6e08403acf2cf2a"
format_stamp: "Formatted at 2026-08-08 04:00:38 on dist-test-slave-pww0"
server_key: "0e395098d03ab8ca09543138bbf16faf"
server_key_iv: "245d68cb816d4b5b26d72103092d3f19"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:38.300048 17784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "9bb7457951c040a2a6e08403acf2cf2a"
format_stamp: "Formatted at 2026-08-08 04:00:38 on dist-test-slave-pww0"
server_key: "0e395098d03ab8ca09543138bbf16faf"
server_key_iv: "245d68cb816d4b5b26d72103092d3f19"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:38.307700 17784 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260808 04:00:38.330307 17811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:38.332391 17784 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:00:38.332651 17784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "9bb7457951c040a2a6e08403acf2cf2a"
format_stamp: "Formatted at 2026-08-08 04:00:38 on dist-test-slave-pww0"
server_key: "0e395098d03ab8ca09543138bbf16faf"
server_key_iv: "245d68cb816d4b5b26d72103092d3f19"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:38.332957 17784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:38.348656 17784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:38.352622 17784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:38.352973 17784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:38.371938 17784 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:38543
I20260808 04:00:38.371970 17898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:38543 every 8 connection(s)
I20260808 04:00:38.373368 17784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:00:38.374904 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 17784
I20260808 04:00:38.375142 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:00:38.376063 29679 external_mini_cluster.cc:1428] Setting key 24137ab2fa1092e0237e1b1291db4585
I20260808 04:00:38.378513 17903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 04:00:38 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:38.392575 17903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a: Bootstrap starting.
I20260808 04:00:38.396062 17903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:38.397405 17903 log.cc:824] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:38.400327 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:38.378355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.28.251.254:38543)
0808 04:00:38.379205 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:38.379218 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:38.379264 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:00:38.380946 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:38.380970 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:38.381398 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:38.381760 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:38.383514 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:38.384961 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:38.386881 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:38.387693 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:38.387834 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:38.392746 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:38.392758 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:38.392775 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:38.392826 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:38.395913 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:38.396817 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:38.396824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:38.396831 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:38.397036 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:38.397474 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:38.397477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:38.397479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:38.398168 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:38.398508 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:38.398894 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 04:00:38.399300 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":426,"threads_started":1}
I20260808 04:00:38.400713 17903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a: No bootstrap required, opened a new log
I20260808 04:00:38.408836 17903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } }
I20260808 04:00:38.409240 17903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:38.409354 17903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb7457951c040a2a6e08403acf2cf2a, State: Initialized, Role: FOLLOWER
I20260808 04:00:38.409906 17903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } }
I20260808 04:00:38.410146 17903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:00:38.410274 17903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:00:38.410460 17903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:38.412796 17903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } }
I20260808 04:00:38.413401 17903 leader_election.cc:302] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bb7457951c040a2a6e08403acf2cf2a; no voters:
I20260808 04:00:38.414101 17903 leader_election.cc:288] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:00:38.414213 17911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:00:38.414944 17911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 1 LEADER]: Becoming Leader. State: Replica: 9bb7457951c040a2a6e08403acf2cf2a, State: Running, Role: LEADER
I20260808 04:00:38.415503 17911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } }
I20260808 04:00:38.416347 17903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:00:38.418617 17913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bb7457951c040a2a6e08403acf2cf2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } } }
I20260808 04:00:38.418705 17914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bb7457951c040a2a6e08403acf2cf2a. Latest consensus state: current_term: 1 leader_uuid: "9bb7457951c040a2a6e08403acf2cf2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } } }
I20260808 04:00:38.419154 17914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [sys.catalog]: This master's current role is: LEADER
I20260808 04:00:38.419389 17913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [sys.catalog]: This master's current role is: LEADER
I20260808 04:00:38.420954 17920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:00:38.426872 17920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:00:38.427587 17903 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:00:38.450389 17920 catalog_manager.cc:1347] Generated new cluster ID: 3311712458cc4740a50ee52c7e1c052f
I20260808 04:00:38.450639 17920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:00:38.464972 17920 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:00:38.466558 17920 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:00:38.476451 17920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a: Generated new TSK 0
I20260808 04:00:38.477468 17920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 04:00:39.303277 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:58079/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:38543
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:00:39 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786161639, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 04:00:39.782948 17970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:39.783414 17970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:39.783614 17970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:39.795390 17970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:39.795643 17970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:39.795895 17970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:00:39.811972 17970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.251.212:58079/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:38543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:39.813925 17970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:39.815821 17970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:39.834570 17990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:39.834589 17992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:39.840953 17970 server_base.cc:1034] running on GCE node
W20260808 04:00:39.841411 17999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:39.842264 17970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:39.843689 17970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:39.845021 17970 hybrid_clock.cc:648] HybridClock initialized: now 1786161639844900 us; error 105 us; skew 500 ppm
I20260808 04:00:39.845463 17970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:39 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161639, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:39.850332 17970 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 04:00:39.860016 17970 webserver.cc:460] Webserver started at http://127.28.251.193:36661/ using document root <none> and password file <none>
I20260808 04:00:39.861241 17970 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:39.861428 17970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:39.861893 17970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:00:39 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161639, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 04:00:39.884945 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "84e8a49fa9e34b7884fe9d7dec8da174"
format_stamp: "Formatted at 2026-08-08 04:00:39 on dist-test-slave-pww0"
server_key: "ef9ea5d48c130f3ce8120cad8e1a1143"
server_key_iv: "cda40c3d49c8bcce834b39cf82c6ae62"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:39.886631 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "84e8a49fa9e34b7884fe9d7dec8da174"
format_stamp: "Formatted at 2026-08-08 04:00:39 on dist-test-slave-pww0"
server_key: "ef9ea5d48c130f3ce8120cad8e1a1143"
server_key_iv: "cda40c3d49c8bcce834b39cf82c6ae62"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:39.894438 17970 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260808 04:00:39.915247 18009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:39.917023 17970 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260808 04:00:39.917243 17970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "84e8a49fa9e34b7884fe9d7dec8da174"
format_stamp: "Formatted at 2026-08-08 04:00:39 on dist-test-slave-pww0"
server_key: "ef9ea5d48c130f3ce8120cad8e1a1143"
server_key_iv: "cda40c3d49c8bcce834b39cf82c6ae62"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:39.917506 17970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:39.943665 17970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:39.949829 17970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:39.950289 17970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:39.953044 17970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:00:39.955344 17970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:00:39.955556 17970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 04:00:39.955780 17970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:00:39.955915 17970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:40.046527 17970 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:33729
I20260808 04:00:40.046576 18189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:33729 every 8 connection(s)
I20260808 04:00:40.048455 17970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:00:40.053759 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 17970
I20260808 04:00:40.053997 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 04:00:40.054832 29679 external_mini_cluster.cc:1428] Setting key c5b48ffea6392516c2382687a4303b69
Aug 08 04:00:40 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161639, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:40.103612 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:40.051730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:40027 (local address 127.28.251.254:38543)
0808 04:00:40.078668 (+ 26938us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:40.078673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:40.079553 (+ 880us) server_negotiation.cc:407] Connection header received
0808 04:00:40.082553 (+ 3000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:40.082559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:40.082677 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:40.082791 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:40.085357 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:40.086566 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:40.089321 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:40.090016 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:40.090132 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:40.095310 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:40.095319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:40.095324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:40.095375 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:40.099189 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:40.100415 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:40.100422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:40.100427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:40.100607 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:40.101221 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:40.101227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:40.101231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:40.101627 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:40.101769 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:40.102893 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0808 04:00:40.103158 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26788,"thread_start_us":26643,"threads_started":1}
I20260808 04:00:40.106209 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:40.052483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:38543 (local address 127.28.251.193:40027)
0808 04:00:40.079329 (+ 26846us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:40.079400 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:40.082039 (+ 2639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:40.083346 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:40.083360 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:40.084129 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:40.085098 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:40.085132 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:40.086818 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:40.086824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:40.089098 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:40.089133 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:40.090569 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:40.090577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:40.090933 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:40.091867 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:40.091903 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:40.094961 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:40.099445 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:40.099492 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:40.099514 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:40.100241 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:40.100778 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:40.100783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:40.100787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:40.101078 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:40.101895 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:40.101904 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:40.102408 (+ 504us) client_negotiation.cc:769] Sending connection context
0808 04:00:40.104129 (+ 1721us) client_negotiation.cc:240] Negotiation successful
0808 04:00:40.104748 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26445,"thread_start_us":26694,"threads_started":1}
Aug 08 04:00:40 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161639, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 04:00:40.117851 18192 heartbeater.cc:347] Connected to a master server at 127.28.251.254:38543
I20260808 04:00:40.118286 18192 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:40.119357 18192 heartbeater.cc:511] Master 127.28.251.254:38543 requested a full tablet report, sending...
I20260808 04:00:40.122294 17834 ts_manager.cc:194] Registered new tserver with Master: 84e8a49fa9e34b7884fe9d7dec8da174 (127.28.251.193:33729)
I20260808 04:00:40.124243 17834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:40027
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
W20260808 04:00:40.197885 29679 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.251.194: real 0.139s user 0.000s sys 0.032s
I20260808 04:00:40.198906 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:58079/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:38543
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:00:40 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161639, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
W20260808 04:00:40.726874 18202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:40.727370 18202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:40.727600 18202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:40.739603 18202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:40.739897 18202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:40.740157 18202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:00:40.756794 18202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.251.212:58079/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:38543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:40.758811 18202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:40.760802 18202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:40.798137 18212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:40.798067 18213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:40.798666 18202 server_base.cc:1034] running on GCE node
W20260808 04:00:40.799309 18219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:40.800284 18202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:40.801682 18202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:40.802968 18202 hybrid_clock.cc:648] HybridClock initialized: now 1786161640802892 us; error 70 us; skew 500 ppm
I20260808 04:00:40.803383 18202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:40 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161640, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:40.807824 18202 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 04:00:40.809887 18202 webserver.cc:460] Webserver started at http://127.28.251.194:34393/ using document root <none> and password file <none>
I20260808 04:00:40.810906 18202 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:40.811064 18202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:40.811451 18202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:00:40 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161640, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 04:00:40.832432 18202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "1c251ae790fb4cc198421eb234585b50"
format_stamp: "Formatted at 2026-08-08 04:00:40 on dist-test-slave-pww0"
server_key: "6eda111c15d98566e82194155daef4d0"
server_key_iv: "d8f90590967accec03cac27ab385ffaf"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:40.833606 18202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "1c251ae790fb4cc198421eb234585b50"
format_stamp: "Formatted at 2026-08-08 04:00:40 on dist-test-slave-pww0"
server_key: "6eda111c15d98566e82194155daef4d0"
server_key_iv: "d8f90590967accec03cac27ab385ffaf"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:40.841740 18202 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.003s
I20260808 04:00:40.863034 18235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:40.865046 18202 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260808 04:00:40.865351 18202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "1c251ae790fb4cc198421eb234585b50"
format_stamp: "Formatted at 2026-08-08 04:00:40 on dist-test-slave-pww0"
server_key: "6eda111c15d98566e82194155daef4d0"
server_key_iv: "d8f90590967accec03cac27ab385ffaf"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:40.865753 18202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:40.894737 18202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:40.900715 18202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:40.901180 18202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:40.902608 18202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:00:40.904636 18202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:00:40.904803 18202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 04:00:40.905021 18202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:00:40.905126 18202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:40.947842 18202 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:37167
I20260808 04:00:40.947891 18408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:37167 every 8 connection(s)
I20260808 04:00:40.949824 18202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:00:40.952374 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 18202
I20260808 04:00:40.952602 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 04:00:40.953303 29679 external_mini_cluster.cc:1428] Setting key 44f03b363ff3af4cc20bbe3f7784defa
Aug 08 04:00:40 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161640, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:40.992646 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:40.956530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:38543 (local address 127.28.251.194:60643)
0808 04:00:40.964376 (+ 7846us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:40.964452 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:40.966952 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:40.971876 (+ 4924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:40.971889 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:40.972572 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:40.973482 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:40.973512 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:40.975146 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:40.975153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:40.977312 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:40.977333 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:40.978690 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:40.978698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:40.979006 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:40.979904 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:40.979938 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:40.982893 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:40.987321 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:40.987333 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:40.987355 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:40.988074 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:40.988594 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:40.988599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:40.988603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:40.988899 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:40.989694 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:40.989704 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:40.990174 (+ 470us) client_negotiation.cc:769] Sending connection context
0808 04:00:40.990524 (+ 350us) client_negotiation.cc:240] Negotiation successful
0808 04:00:40.991131 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7384,"thread_start_us":10798,"threads_started":1}
I20260808 04:00:40.993870 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:40.955663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:60643 (local address 127.28.251.254:38543)
0808 04:00:40.970508 (+ 14845us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:40.970515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:40.970543 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:00:40.970620 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:40.970626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:40.970716 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:40.970880 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:40.973744 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:40.974904 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:40.977545 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:40.978192 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:40.978337 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:40.983231 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:40.983238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:40.983243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:40.983302 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:40.987070 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:40.988254 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:40.988260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:40.988264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:40.988450 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:40.989057 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:40.989063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:40.989066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:40.989466 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:40.989612 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:40.993192 (+ 3580us) server_negotiation.cc:299] Negotiation successful
0808 04:00:40.993491 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14638,"thread_start_us":15313,"threads_started":1}
I20260808 04:00:40.996493 18409 heartbeater.cc:347] Connected to a master server at 127.28.251.254:38543
I20260808 04:00:40.996909 18409 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:40.997998 18409 heartbeater.cc:511] Master 127.28.251.254:38543 requested a full tablet report, sending...
I20260808 04:00:40.999977 17834 ts_manager.cc:194] Registered new tserver with Master: 1c251ae790fb4cc198421eb234585b50 (127.28.251.194:37167)
I20260808 04:00:41.001180 17834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:60643
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 04:00:41.051265 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:58079/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:38543
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:00:41.128331 18192 heartbeater.cc:503] Master 127.28.251.254:38543 was elected leader, sending a full tablet report...
W20260808 04:00:41.492331 18418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:41.492645 18418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:41.492751 18418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:41.500370 18418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:41.500511 18418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:41.500643 18418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:00:41.510846 18418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.251.212:58079/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:38543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:41.512264 18418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:41.513592 18418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:41.527654 18425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:41.527690 18426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:41.527729 18428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:41.528997 18418 server_base.cc:1034] running on GCE node
I20260808 04:00:41.529667 18418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:41.530710 18418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:41.531919 18418 hybrid_clock.cc:648] HybridClock initialized: now 1786161641531851 us; error 57 us; skew 500 ppm
I20260808 04:00:41.532231 18418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:41 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161641, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:41.535528 18418 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 04:00:41.537287 18418 webserver.cc:460] Webserver started at http://127.28.251.195:39393/ using document root <none> and password file <none>
I20260808 04:00:41.538012 18418 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:41.538122 18418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:41.538460 18418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:00:41 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161641, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 04:00:41.557158 18418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "326a2250ef33472f8b7c430ea003fd34"
format_stamp: "Formatted at 2026-08-08 04:00:41 on dist-test-slave-pww0"
server_key: "0fb09aa6195ea1b04b76725b39d00b29"
server_key_iv: "d0df4679487d4b964bdb908f83497882"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:41.558107 18418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "326a2250ef33472f8b7c430ea003fd34"
format_stamp: "Formatted at 2026-08-08 04:00:41 on dist-test-slave-pww0"
server_key: "0fb09aa6195ea1b04b76725b39d00b29"
server_key_iv: "d0df4679487d4b964bdb908f83497882"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:41.564055 18418 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260808 04:00:41.582803 18438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:41.584702 18418 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:00:41.584931 18418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "326a2250ef33472f8b7c430ea003fd34"
format_stamp: "Formatted at 2026-08-08 04:00:41 on dist-test-slave-pww0"
server_key: "0fb09aa6195ea1b04b76725b39d00b29"
server_key_iv: "d0df4679487d4b964bdb908f83497882"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:41.585209 18418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:41.610340 18418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:41.614691 18418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:41.614985 18418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:41.616087 18418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:00:41.617764 18418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:00:41.617861 18418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:41.617964 18418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:00:41.618008 18418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:41.650792 18418 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:43923
I20260808 04:00:41.650820 18635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:43923 every 8 connection(s)
I20260808 04:00:41.652215 18418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:00:41.662076 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 18418
I20260808 04:00:41.662243 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 04:00:41.662952 29679 external_mini_cluster.cc:1428] Setting key 259ab08c33748b9a615c587113fa2103
Aug 08 04:00:41 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161641, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:41.672736 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:41.654920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:42147 (local address 127.28.251.254:38543)
0808 04:00:41.655510 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:41.655517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:41.656530 (+ 1013us) server_negotiation.cc:407] Connection header received
0808 04:00:41.658315 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:41.658319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:41.658414 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:41.658515 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:41.660014 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:41.660761 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:41.662469 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:41.663203 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:41.663374 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:41.667027 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:41.667032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:41.667036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:41.667070 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:41.669780 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:41.670522 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:41.670525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:41.670527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:41.670657 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:41.671069 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:41.671073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:41.671075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:41.671353 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:41.671531 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:41.672219 (+ 688us) server_negotiation.cc:299] Negotiation successful
0808 04:00:41.672424 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":243,"threads_started":1}
I20260808 04:00:41.673738 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:41.655532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:38543 (local address 127.28.251.195:42147)
0808 04:00:41.656302 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:41.656397 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:41.658048 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:41.658718 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:41.658727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:41.659170 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:41.659852 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:41.659871 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:41.660891 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:41.660895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:41.662258 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:41.662282 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:41.663446 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:41.663451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:41.663773 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:41.664512 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:41.664553 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:41.666786 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:41.669927 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:41.669935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:41.669951 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:41.670423 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:41.670746 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:41.670749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:41.670752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:41.670979 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:41.671447 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:41.671453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:41.671866 (+ 413us) client_negotiation.cc:769] Sending connection context
0808 04:00:41.672113 (+ 247us) client_negotiation.cc:240] Negotiation successful
0808 04:00:41.672529 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":293,"threads_started":1}
I20260808 04:00:41.675175 18636 heartbeater.cc:347] Connected to a master server at 127.28.251.254:38543
I20260808 04:00:41.675518 18636 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:41.676258 18636 heartbeater.cc:511] Master 127.28.251.254:38543 requested a full tablet report, sending...
I20260808 04:00:41.677908 17834 ts_manager.cc:194] Registered new tserver with Master: 326a2250ef33472f8b7c430ea003fd34 (127.28.251.195:43923)
I20260808 04:00:41.678886 17834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:42147
I20260808 04:00:41.678948 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:00:41.680719 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 17970
Aug 08 04:00:41 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 04:00:41.697523 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:41.681704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59828 (local address 127.28.251.193:33729)
0808 04:00:41.682310 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:41.682316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:41.682349 (+ 33us) server_negotiation.cc:407] Connection header received
0808 04:00:41.682407 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:41.682413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:41.682597 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:41.682728 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:41.684337 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:41.685362 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:41.686729 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:41.687287 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:41.687395 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:41.691874 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:41.691889 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:41.691905 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:41.691951 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:41.694616 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:41.695378 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:41.695386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:41.695389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:41.695605 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:41.695996 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:41.696001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:41.696004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:41.696379 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:41.696669 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:41.696933 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 04:00:41.697151 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":292,"threads_started":1}
I20260808 04:00:41.698889 18180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:41.786095 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 17970
I20260808 04:00:41.813112 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 18202
Aug 08 04:00:41 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 04:00:41.830168 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:41.814044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.28.251.194:37167)
0808 04:00:41.814595 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:41.814602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:41.814629 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:00:41.814692 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:41.814698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:41.814932 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:41.815080 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:41.816476 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:41.817612 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:41.818943 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:41.819555 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:41.819670 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:41.824276 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:41.824289 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:41.824304 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:41.824345 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:41.827096 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:41.827899 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:41.827907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:41.827909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:41.828053 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:41.828452 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:41.828456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:41.828458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:41.828839 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:41.829178 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:41.829507 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 04:00:41.829811 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":263,"threads_started":1}
I20260808 04:00:41.831506 18403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:41.916222 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 18202
I20260808 04:00:41.948715 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 18418
Aug 08 04:00:41 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 04:00:41.965790 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:41.949725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49670 (local address 127.28.251.195:43923)
0808 04:00:41.950421 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:41.950428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:41.950459 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:00:41.950522 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:41.950529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:41.950747 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:41.950909 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:41.952377 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:41.953431 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:41.954709 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:41.955296 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:41.955429 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:41.959975 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:41.959989 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:41.960006 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:41.960048 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:41.962774 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:41.963550 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:41.963558 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:41.963561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:41.963706 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:41.964108 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:41.964111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:41.964113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:41.964505 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:41.964830 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:41.965148 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 04:00:41.965435 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":337,"threads_started":1}
I20260808 04:00:41.967211 18629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:42.051991 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 18418
I20260808 04:00:42.072192 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 17784
I20260808 04:00:42.073583 17888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:42.185108 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 17784
I20260808 04:00:42.222015 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:38543
--webserver_interface=127.28.251.254
--webserver_port=43861
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:38543
--ranger_config_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:58079/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.251.212:58079/kms with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:00:42.588184 18656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:42.588528 18656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:42.588685 18656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:42.597155 18656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:00:42.597301 18656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:42.597388 18656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:42.597479 18656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:00:42.597559 18656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:00:42.611109 18656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.251.212:58079/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:38543
--ranger_config_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:38543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=43861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:42.613149 18656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:42.615648 18656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:42.631546 18667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:42.633040 18663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:42.634316 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:42.635041 18656 server_base.cc:1034] running on GCE node
I20260808 04:00:42.636106 18656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:42.638402 18656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:42.639631 18656 hybrid_clock.cc:648] HybridClock initialized: now 1786161642639569 us; error 67 us; skew 500 ppm
I20260808 04:00:42.640056 18656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:42 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161642, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:42.644351 18656 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:00:42.646229 18656 webserver.cc:460] Webserver started at http://127.28.251.254:43861/ using document root <none> and password file <none>
I20260808 04:00:42.647014 18656 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:42.647154 18656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:00:42 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161642, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 04:00:42.690318 18656 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:00:42.693813 18684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:42.695708 18656 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:00:42.695935 18656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "9bb7457951c040a2a6e08403acf2cf2a"
format_stamp: "Formatted at 2026-08-08 04:00:38 on dist-test-slave-pww0"
server_key: "0e395098d03ab8ca09543138bbf16faf"
server_key_iv: "245d68cb816d4b5b26d72103092d3f19"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:42.696967 18656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:42.710649 18656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:42.714589 18656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:42.714895 18656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:42.733650 18656 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:38543
I20260808 04:00:42.733671 18780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:38543 every 8 connection(s)
I20260808 04:00:42.735201 18656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:00:42.738667 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 18656
I20260808 04:00:42.739848 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:58079/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:33729
--local_ip_for_outbound_sockets=127.28.251.193
--tserver_master_addrs=127.28.251.254:38543
--webserver_port=36661
--webserver_interface=127.28.251.193
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.251.212:58079/kms with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:00:42.744261 18782 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:00:42.748976 18782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a: Bootstrap starting.
I20260808 04:00:42.770219 18782 log.cc:824] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:42.780174 18782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:00:42.780750 18782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a: Bootstrap complete.
I20260808 04:00:42.790017 18782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } }
I20260808 04:00:42.790918 18782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb7457951c040a2a6e08403acf2cf2a, State: Initialized, Role: FOLLOWER
I20260808 04:00:42.791630 18782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } }
I20260808 04:00:42.791882 18782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:00:42.791994 18782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:00:42.792161 18782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:00:42.796290 18782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } }
I20260808 04:00:42.796802 18782 leader_election.cc:302] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bb7457951c040a2a6e08403acf2cf2a; no voters:
I20260808 04:00:42.797433 18782 leader_election.cc:288] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:00:42.797657 18786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:00:42.798621 18786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [term 2 LEADER]: Becoming Leader. State: Replica: 9bb7457951c040a2a6e08403acf2cf2a, State: Running, Role: LEADER
I20260808 04:00:42.799412 18786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } }
I20260808 04:00:42.799635 18782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:00:42.806877 18789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bb7457951c040a2a6e08403acf2cf2a. Latest consensus state: current_term: 2 leader_uuid: "9bb7457951c040a2a6e08403acf2cf2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } } }
I20260808 04:00:42.807341 18789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [sys.catalog]: This master's current role is: LEADER
I20260808 04:00:42.807898 18787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9bb7457951c040a2a6e08403acf2cf2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb7457951c040a2a6e08403acf2cf2a" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 38543 } } }
I20260808 04:00:42.808159 18787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a [sys.catalog]: This master's current role is: LEADER
I20260808 04:00:42.816499 18809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:00:42.822494 18809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:00:42.829444 18782 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:00:42.830178 18809 catalog_manager.cc:1259] Loaded cluster ID: 3311712458cc4740a50ee52c7e1c052f
I20260808 04:00:42.830418 18809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:00:42.834851 18809 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:00:42.836596 18809 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a: Loaded TSK: 0
I20260808 04:00:42.838375 18809 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:00:43.220614 18784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:43.221149 18784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:43.221416 18784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:43.233508 18784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:43.233747 18784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:43.233989 18784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:00:43.250572 18784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.251.212:58079/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:33729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=36661
--tserver_master_addrs=127.28.251.254:38543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:43.252882 18784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:43.254966 18784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:43.287520 18846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:43.287362 18847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:43.304922 18849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:43.305243 18784 server_base.cc:1034] running on GCE node
I20260808 04:00:43.306449 18784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:43.307782 18784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:43.309144 18784 hybrid_clock.cc:648] HybridClock initialized: now 1786161643308979 us; error 150 us; skew 500 ppm
I20260808 04:00:43.309654 18784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:43 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161643, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:43.318377 18784 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 04:00:43.320912 18784 webserver.cc:460] Webserver started at http://127.28.251.193:36661/ using document root <none> and password file <none>
I20260808 04:00:43.321736 18784 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:43.321902 18784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:00:43 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161643, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 04:00:43.347522 18784 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:00:43.351315 18863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:43.353382 18784 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:00:43.353613 18784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "84e8a49fa9e34b7884fe9d7dec8da174"
format_stamp: "Formatted at 2026-08-08 04:00:39 on dist-test-slave-pww0"
server_key: "ef9ea5d48c130f3ce8120cad8e1a1143"
server_key_iv: "cda40c3d49c8bcce834b39cf82c6ae62"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:43.355254 18784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:43.392823 18784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:43.398047 18784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:43.398409 18784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:43.399818 18784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:00:43.402081 18784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:00:43.402232 18784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 04:00:43.402343 18784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:00:43.402415 18784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:43.460251 18784 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:33729
I20260808 04:00:43.460295 19047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:33729 every 8 connection(s)
I20260808 04:00:43.462236 18784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:00:43.468485 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 18784
I20260808 04:00:43.469367 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:58079/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:37167
--local_ip_for_outbound_sockets=127.28.251.194
--tserver_master_addrs=127.28.251.254:38543
--webserver_port=34393
--webserver_interface=127.28.251.194
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.251.212:58079/kms with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:00:43 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161643, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:43.498149 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:43.465919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:32923 (local address 127.28.251.254:38543)
0808 04:00:43.467895 (+ 1976us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:43.467914 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:43.467968 (+ 54us) server_negotiation.cc:407] Connection header received
0808 04:00:43.473182 (+ 5214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:43.473201 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:43.473750 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:43.474197 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:43.476881 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:43.478740 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:43.481345 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:43.482388 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:43.482594 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:43.487298 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:43.487311 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:43.487333 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:43.487391 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:43.491562 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:43.492723 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:43.492735 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:43.492744 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:43.493020 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:43.493675 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:43.493681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:43.493684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:43.494498 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:43.494756 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:43.496202 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0808 04:00:43.496678 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1691,"thread_start_us":1452,"threads_started":1}
I20260808 04:00:43.498350 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:43.466024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:38543 (local address 127.28.251.193:32923)
0808 04:00:43.466891 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:43.466940 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:43.472211 (+ 5271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:43.474880 (+ 2669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:43.474893 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:43.475654 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:43.476543 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:43.476576 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:43.478981 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:43.478987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:43.480955 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:43.480976 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:43.483027 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:43.483035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:43.483340 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:43.484191 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:43.484234 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:43.486894 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:43.491774 (+ 4880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:43.491786 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:43.491807 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:43.492494 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:43.493182 (+ 688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:43.493187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:43.493190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:43.493506 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:43.494870 (+ 1364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:43.494879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:43.495539 (+ 660us) client_negotiation.cc:769] Sending connection context
0808 04:00:43.495861 (+ 322us) client_negotiation.cc:240] Negotiation successful
0808 04:00:43.496496 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":354,"threads_started":1}
I20260808 04:00:43.502292 19048 heartbeater.cc:347] Connected to a master server at 127.28.251.254:38543
I20260808 04:00:43.502679 19048 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:43.503710 19048 heartbeater.cc:511] Master 127.28.251.254:38543 requested a full tablet report, sending...
W20260808 04:00:43.507789 19048 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:38543 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:00:43.508056 19048 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:43.508383 19048 heartbeater.cc:511] Master 127.28.251.254:38543 requested a full tablet report, sending...
I20260808 04:00:43.509950 19048 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:43.510391 19048 heartbeater.cc:511] Master 127.28.251.254:38543 requested a full tablet report, sending...
W20260808 04:00:43.511807 19048 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:00:43.906189 19054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:43.906498 19054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:43.906646 19054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:43.916100 19054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:43.916253 19054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:43.916487 19054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:00:43.930091 19054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.251.212:58079/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:37167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=34393
--tserver_master_addrs=127.28.251.254:38543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:43.931957 19054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:43.933697 19054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:43.951947 19082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:43.956502 19080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:43.956818 19054 server_base.cc:1034] running on GCE node
W20260808 04:00:43.957183 19087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:43.958587 19054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:43.959761 19054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:43.961076 19054 hybrid_clock.cc:648] HybridClock initialized: now 1786161643961005 us; error 65 us; skew 500 ppm
I20260808 04:00:43.961503 19054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:43 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161643, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:43.965190 19054 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 04:00:43.969841 19054 webserver.cc:460] Webserver started at http://127.28.251.194:34393/ using document root <none> and password file <none>
I20260808 04:00:43.970939 19054 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:43.971130 19054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:00:43 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161643, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 04:00:44.006033 19054 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260808 04:00:44.009663 19099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:44.011946 19054 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 04:00:44.012200 19054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "1c251ae790fb4cc198421eb234585b50"
format_stamp: "Formatted at 2026-08-08 04:00:40 on dist-test-slave-pww0"
server_key: "6eda111c15d98566e82194155daef4d0"
server_key_iv: "d8f90590967accec03cac27ab385ffaf"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:44.013370 19054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:44.063752 19054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:44.072633 19054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:44.072994 19054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:44.074417 19054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:00:44.076396 19054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:00:44.076514 19054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:44.076639 19054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:00:44.076725 19054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:00:44 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786161644, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:44.165781 19054 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:37167
I20260808 04:00:44.165835 19288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:37167 every 8 connection(s)
I20260808 04:00:44.167331 19054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:00:44.170593 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 19054
I20260808 04:00:44.171422 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.251.212:58079/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:43923
--local_ip_for_outbound_sockets=127.28.251.195
--tserver_master_addrs=127.28.251.254:38543
--webserver_port=39393
--webserver_interface=127.28.251.195
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.251.212:58079/kms with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:00:44 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161643, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:44.210721 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.171816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:34389 (local address 127.28.251.254:38543)
0808 04:00:44.187077 (+ 15261us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:44.187085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:44.188693 (+ 1608us) server_negotiation.cc:407] Connection header received
0808 04:00:44.190799 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:44.190806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:44.190897 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:44.190977 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:44.193358 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.194426 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.196921 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.197565 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.197692 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:44.202568 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:44.202574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:44.202578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:44.202635 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:44.206295 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:44.207491 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:44.207498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:44.207502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:44.207672 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:44.208303 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:44.208308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:44.208323 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:44.208730 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:44.208882 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:44.210042 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0808 04:00:44.210305 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15091,"thread_start_us":15403,"threads_started":1}
I20260808 04:00:44.213691 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.171262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:38543 (local address 127.28.251.194:34389)
0808 04:00:44.188488 (+ 17226us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:44.188548 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:44.190461 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:44.191619 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:44.191631 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:44.192251 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:44.193110 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:44.193137 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.194655 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.194660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:44.196727 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:44.196744 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.198086 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.198093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:44.198390 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:44.199269 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:44.199364 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:44.202263 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:44.206538 (+ 4275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:44.206549 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:44.206571 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:44.207267 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:44.207843 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:44.207848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:44.207852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:44.208154 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:44.208998 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:44.209007 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:44.209552 (+ 545us) client_negotiation.cc:769] Sending connection context
0808 04:00:44.211385 (+ 1833us) client_negotiation.cc:240] Negotiation successful
0808 04:00:44.212102 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16866,"thread_start_us":16614,"threads_started":1}
I20260808 04:00:44.216156 19290 heartbeater.cc:347] Connected to a master server at 127.28.251.254:38543
I20260808 04:00:44.216506 19290 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:44.217293 19290 heartbeater.cc:511] Master 127.28.251.254:38543 requested a full tablet report, sending...
I20260808 04:00:44.219396 18721 ts_manager.cc:194] Registered new tserver with Master: 1c251ae790fb4cc198421eb234585b50 (127.28.251.194:37167)
I20260808 04:00:44.221730 18721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:34389
I20260808 04:00:44.511963 19048 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:44.512462 19048 heartbeater.cc:511] Master 127.28.251.254:38543 requested a full tablet report, sending...
I20260808 04:00:44.513974 18721 ts_manager.cc:194] Registered new tserver with Master: 84e8a49fa9e34b7884fe9d7dec8da174 (127.28.251.193:33729)
I20260808 04:00:44.515156 18721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:32923
Aug 08 04:00:44 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161644, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
W20260808 04:00:44.622567 19295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:44.623152 19295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:44.623389 19295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:44.635155 19295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:44.635964 19295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:44.636317 19295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:00:44.652607 19295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.251.212:58079/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:43923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=39393
--tserver_master_addrs=127.28.251.254:38543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:44.654980 19295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:44.657034 19295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:44.675170 19301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:44.675159 19302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:44.681252 19295 server_base.cc:1034] running on GCE node
W20260808 04:00:44.681936 19310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:44.682716 19295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:44.684018 19295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:44.685411 19295 hybrid_clock.cc:648] HybridClock initialized: now 1786161644685334 us; error 56 us; skew 500 ppm
I20260808 04:00:44.685809 19295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:44 dist-test-slave-pww0 krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161644, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:44.689150 19295 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 04:00:44.691042 19295 webserver.cc:460] Webserver started at http://127.28.251.195:39393/ using document root <none> and password file <none>
I20260808 04:00:44.691879 19295 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:44.692030 19295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:00:44 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161644, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 04:00:44.715342 19295 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:00:44.719259 19327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:44.721221 19295 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:00:44.721451 19295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "326a2250ef33472f8b7c430ea003fd34"
format_stamp: "Formatted at 2026-08-08 04:00:41 on dist-test-slave-pww0"
server_key: "0fb09aa6195ea1b04b76725b39d00b29"
server_key_iv: "d0df4679487d4b964bdb908f83497882"
server_key_version: "kuduclusterkey@0"
I20260808 04:00:44.722937 19295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:44.746030 19295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:44.751718 19295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:44.752123 19295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:44.755944 19295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:00:44.760484 19295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:00:44.760618 19295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:44.760784 19295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:00:44.760870 19295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:44.813800 19295 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:43923
I20260808 04:00:44.813901 19527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:43923 every 8 connection(s)
I20260808 04:00:44.815912 19295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:00:44.825515 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 19295
Aug 08 04:00:44 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161644, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:44.866763 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.819184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:57591 (local address 127.28.251.254:38543)
0808 04:00:44.846022 (+ 26838us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:44.846030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:44.846056 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:00:44.846135 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:44.846141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:44.846233 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:44.846367 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:44.849671 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.850948 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.853404 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.854097 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.854188 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:44.858628 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:44.858634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:44.858638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:44.858690 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:44.862384 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:44.863552 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:44.863559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:44.863562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:44.863736 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:44.864393 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:44.864400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:44.864404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:44.864793 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:44.864948 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:44.866091 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0808 04:00:44.866359 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26678,"thread_start_us":28188,"threads_started":1}
I20260808 04:00:44.867878 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.826785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.28.251.254:38543)
0808 04:00:44.846988 (+ 20203us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:44.846995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:44.847014 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:00:44.847073 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:44.847077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:44.847141 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:44.847239 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:44.849501 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.850737 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.853076 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.853785 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.853906 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:44.858277 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:44.858285 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:44.858290 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:44.858339 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:44.863554 (+ 5215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:44.864593 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:44.864598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:44.864602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:44.864779 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:44.865277 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:44.865282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:44.865285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:44.865666 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:44.865949 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:44.867235 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0808 04:00:44.867496 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20071,"thread_start_us":19723,"threads_started":1}
I20260808 04:00:44.870256 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.819888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:38543 (local address 127.28.251.195:57591)
0808 04:00:44.842343 (+ 22455us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:44.842414 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:44.845052 (+ 2638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:44.847906 (+ 2854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:44.847921 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:44.848559 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:44.849417 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:44.849445 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.851169 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.851173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:44.853203 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:44.853223 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.854555 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.854561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:44.854880 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:44.855574 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:44.855605 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:44.858307 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:44.862604 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:44.862615 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:44.862638 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:44.863316 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:44.863909 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:44.863914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:44.863917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:44.864225 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:44.865079 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:44.865087 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:44.865578 (+ 491us) client_negotiation.cc:769] Sending connection context
0808 04:00:44.867346 (+ 1768us) client_negotiation.cc:240] Negotiation successful
0808 04:00:44.868475 (+ 1129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22043,"thread_start_us":21831,"threads_started":1}
I20260808 04:00:44.872220 19529 heartbeater.cc:347] Connected to a master server at 127.28.251.254:38543
I20260808 04:00:44.872570 19529 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:44.873493 19529 heartbeater.cc:511] Master 127.28.251.254:38543 requested a full tablet report, sending...
I20260808 04:00:44.875273 18721 ts_manager.cc:194] Registered new tserver with Master: 326a2250ef33472f8b7c430ea003fd34 (127.28.251.195:43923)
I20260808 04:00:44.876472 18721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:57591
I20260808 04:00:44.884047 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:00:44.909965 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.888153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.28.251.254:38543)
0808 04:00:44.888386 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:44.888393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:44.891935 (+ 3542us) server_negotiation.cc:407] Connection header received
0808 04:00:44.892150 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:44.892156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:44.892243 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:44.892369 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:44.894515 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.895706 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.897905 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:44.898584 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:44.898695 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:44.902887 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:44.902894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:44.902899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:44.902949 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:44.906556 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:44.907630 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:44.907636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:44.907639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:44.907822 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:44.908360 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:44.908366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:44.908369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:44.908690 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:44.908947 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:44.909319 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 04:00:44.909560 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:00:44.915939 18721 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:00:44.928774 18721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 08 04:00:44 dist-test-slave-pww0 krb5kdc[17024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161644, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for HTTP/127.28.251.212@KRBTEST.COM
I20260808 04:00:45.013723 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.991821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:43923 (local address 127.0.0.1:49682)
0808 04:00:44.995572 (+ 3751us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:44.995597 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:44.995884 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:44.998191 (+ 2307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:44.998197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:44.998229 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:44.998604 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:44.998621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.003822 (+ 5201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.003831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:45.006460 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:45.006492 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.012456 (+ 5964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.012465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:45.012761 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.012831 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 04:00:45.013008 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 04:00:45.013272 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3566,"thread_start_us":4184,"threads_started":1}
I20260808 04:00:45.016999 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.984982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.28.251.195:43923)
0808 04:00:44.994571 (+ 9589us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:44.994581 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:44.996791 (+ 2210us) server_negotiation.cc:407] Connection header received
0808 04:00:44.996933 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:44.996942 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:44.997318 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:44.997558 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:44.999213 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.003529 (+ 4316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.007314 (+ 3785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.009161 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.009446 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.009664 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:45.016371 (+ 6707us) server_negotiation.cc:299] Negotiation successful
0808 04:00:45.016608 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9342,"thread_start_us":20395,"threads_started":1}
I20260808 04:00:45.022264 19415 tablet_service.cc:1467] Processing CreateTablet for tablet 34133c04c7b448ef887a61783b34dab0 (DEFAULT_TABLE table=test-table [id=6f86207fc96b4d6091968745e7dc8a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:00:45.024360 19415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34133c04c7b448ef887a61783b34dab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:00:45.034327 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.966813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33729 (local address 127.0.0.1:41904)
0808 04:00:44.980883 (+ 14070us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:44.980956 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:44.981227 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:44.992196 (+ 10969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:44.992205 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:44.992274 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:44.992784 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:44.992802 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.026439 (+ 33637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.026446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:45.028553 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:45.028577 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.033373 (+ 4796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.033381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:45.033583 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.033625 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 04:00:45.033727 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 04:00:45.033932 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13647,"thread_start_us":16193,"threads_started":1}
I20260808 04:00:45.035800 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.966894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.28.251.193:33729)
0808 04:00:44.982028 (+ 15134us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:44.982039 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:44.982073 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:00:44.982187 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:44.982196 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:44.982516 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:44.982764 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:45.015311 (+ 32547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.021644 (+ 6333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.028934 (+ 7290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.030496 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.034880 (+ 4384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.035104 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:45.035206 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 04:00:45.035406 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14947,"thread_start_us":14765,"threads_started":1}
I20260808 04:00:45.040376 18940 tablet_service.cc:1467] Processing CreateTablet for tablet 34133c04c7b448ef887a61783b34dab0 (DEFAULT_TABLE table=test-table [id=6f86207fc96b4d6091968745e7dc8a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:00:45.042093 18940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34133c04c7b448ef887a61783b34dab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:00:45.045748 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.996536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:37167 (local address 127.0.0.1:48696)
0808 04:00:45.010196 (+ 13660us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:45.010222 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:45.010426 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:45.035327 (+ 24901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:45.035333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:45.035366 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:45.035775 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:45.035789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.039621 (+ 3832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.039628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:45.041756 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:45.041778 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.044876 (+ 3098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.044883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:45.045073 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.045114 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 04:00:45.045214 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:00:45.045374 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13492,"thread_start_us":13986,"threads_started":1}
I20260808 04:00:45.047014 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:44.996837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.28.251.194:37167)
0808 04:00:45.017200 (+ 20363us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:45.017209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:45.017244 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:00:45.034693 (+ 17449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:45.034724 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:45.035064 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:45.036225 (+ 1161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:45.036507 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.037965 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.042129 (+ 4164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.043881 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.046235 (+ 2354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.046441 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:45.046522 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 04:00:45.046704 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20164,"thread_start_us":20637,"threads_started":1}
I20260808 04:00:45.051563 19179 tablet_service.cc:1467] Processing CreateTablet for tablet 34133c04c7b448ef887a61783b34dab0 (DEFAULT_TABLE table=test-table [id=6f86207fc96b4d6091968745e7dc8a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:00:45.053272 19179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34133c04c7b448ef887a61783b34dab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:00:45.078472 19562 tablet_bootstrap.cc:492] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174: Bootstrap starting.
I20260808 04:00:45.084193 19562 tablet_bootstrap.cc:654] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:45.084358 19561 tablet_bootstrap.cc:492] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: Bootstrap starting.
I20260808 04:00:45.086532 19562 log.cc:824] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:45.087947 19561 tablet_bootstrap.cc:654] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:45.092505 19561 log.cc:824] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:45.098533 19567 tablet_bootstrap.cc:492] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50: Bootstrap starting.
I20260808 04:00:45.103421 19561 tablet_bootstrap.cc:492] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: No bootstrap required, opened a new log
I20260808 04:00:45.103924 19561 ts_tablet_manager.cc:1397] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.003s
I20260808 04:00:45.114863 19562 tablet_bootstrap.cc:492] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174: No bootstrap required, opened a new log
I20260808 04:00:45.115326 19562 ts_tablet_manager.cc:1397] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.009s
I20260808 04:00:45.115896 19567 tablet_bootstrap.cc:654] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:45.117358 19567 log.cc:824] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:45.122946 19561 raft_consensus.cc:348] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33729 } } peers { permanent_uuid: "326a2250ef33472f8b7c430ea003fd34" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43923 } } peers { permanent_uuid: "1c251ae790fb4cc198421eb234585b50" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37167 } }
I20260808 04:00:45.123572 19561 raft_consensus.cc:374] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:45.123728 19561 raft_consensus.cc:729] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326a2250ef33472f8b7c430ea003fd34, State: Initialized, Role: FOLLOWER
I20260808 04:00:45.124481 19561 consensus_queue.cc:260] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33729 } } peers { permanent_uuid: "326a2250ef33472f8b7c430ea003fd34" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43923 } } peers { permanent_uuid: "1c251ae790fb4cc198421eb234585b50" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37167 } }
I20260808 04:00:45.128407 19567 tablet_bootstrap.cc:492] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50: No bootstrap required, opened a new log
I20260808 04:00:45.128744 19567 ts_tablet_manager.cc:1397] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.000s
I20260808 04:00:45.140344 19562 raft_consensus.cc:348] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33729 } } peers { permanent_uuid: "326a2250ef33472f8b7c430ea003fd34" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43923 } } peers { permanent_uuid: "1c251ae790fb4cc198421eb234585b50" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37167 } }
I20260808 04:00:45.141009 19562 raft_consensus.cc:374] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:45.141156 19562 raft_consensus.cc:729] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e8a49fa9e34b7884fe9d7dec8da174, State: Initialized, Role: FOLLOWER
I20260808 04:00:45.142081 19562 consensus_queue.cc:260] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33729 } } peers { permanent_uuid: "326a2250ef33472f8b7c430ea003fd34" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43923 } } peers { permanent_uuid: "1c251ae790fb4cc198421eb234585b50" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37167 } }
I20260808 04:00:45.148414 19529 heartbeater.cc:503] Master 127.28.251.254:38543 was elected leader, sending a full tablet report...
I20260808 04:00:45.154268 19561 ts_tablet_manager.cc:1428] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: Time spent starting tablet: real 0.047s user 0.016s sys 0.001s
I20260808 04:00:45.158202 19048 heartbeater.cc:503] Master 127.28.251.254:38543 was elected leader, sending a full tablet report...
I20260808 04:00:45.159092 19562 ts_tablet_manager.cc:1428] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174: Time spent starting tablet: real 0.043s user 0.014s sys 0.001s
I20260808 04:00:45.166635 19567 raft_consensus.cc:348] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33729 } } peers { permanent_uuid: "326a2250ef33472f8b7c430ea003fd34" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43923 } } peers { permanent_uuid: "1c251ae790fb4cc198421eb234585b50" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37167 } }
I20260808 04:00:45.167290 19567 raft_consensus.cc:374] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:45.167506 19567 raft_consensus.cc:729] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c251ae790fb4cc198421eb234585b50, State: Initialized, Role: FOLLOWER
I20260808 04:00:45.168458 19567 consensus_queue.cc:260] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33729 } } peers { permanent_uuid: "326a2250ef33472f8b7c430ea003fd34" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43923 } } peers { permanent_uuid: "1c251ae790fb4cc198421eb234585b50" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37167 } }
I20260808 04:00:45.176003 19290 heartbeater.cc:503] Master 127.28.251.254:38543 was elected leader, sending a full tablet report...
I20260808 04:00:45.177287 19567 ts_tablet_manager.cc:1428] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50: Time spent starting tablet: real 0.048s user 0.009s sys 0.012s
W20260808 04:00:45.217923 19049 tablet.cc:2369] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:00:45.315888 19570 raft_consensus.cc:482] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:00:45.316216 19570 raft_consensus.cc:504] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33729 } } peers { permanent_uuid: "326a2250ef33472f8b7c430ea003fd34" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43923 } } peers { permanent_uuid: "1c251ae790fb4cc198421eb234585b50" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37167 } }
I20260808 04:00:45.318003 19570 leader_election.cc:288] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84e8a49fa9e34b7884fe9d7dec8da174 (127.28.251.193:33729), 1c251ae790fb4cc198421eb234585b50 (127.28.251.194:37167)
W20260808 04:00:45.319692 19530 tablet.cc:2369] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:00:45.324522 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.318427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33729 (local address 127.28.251.195:49709)
0808 04:00:45.318623 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:45.318661 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:45.318795 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:45.319342 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:45.319350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:45.319382 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:45.319604 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:45.319616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.320882 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.320886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:45.322508 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:45.322524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.323772 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.323778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:45.323980 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.324005 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:00:45.324093 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 04:00:45.324232 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 04:00:45.324522 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.318455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:49709 (local address 127.28.251.193:33729)
0808 04:00:45.318694 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:45.318701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:45.318727 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:00:45.318968 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:45.318974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:45.319056 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:45.319142 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:45.319823 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.320657 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.322690 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.323604 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.323795 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.323880 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:45.324135 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 04:00:45.324271 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:00:45.325515 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.319171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:58599 (local address 127.28.251.194:37167)
0808 04:00:45.319380 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:45.319386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:45.319672 (+ 286us) server_negotiation.cc:407] Connection header received
0808 04:00:45.319984 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:45.319990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:45.320087 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:45.320152 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:45.321039 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.321763 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.323725 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.324554 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.324708 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.324784 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:45.325202 (+ 418us) server_negotiation.cc:299] Negotiation successful
0808 04:00:45.325314 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 04:00:45.325641 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.318927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:37167 (local address 127.28.251.195:58599)
0808 04:00:45.319562 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:45.319580 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:45.319786 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:45.320378 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:45.320382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:45.320409 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:45.320877 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:45.320891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.321920 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.321926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:45.323550 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:45.323565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.324912 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.324917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:45.325039 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.325063 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:00:45.325158 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 04:00:45.325290 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":271,"threads_started":1}
I20260808 04:00:45.325695 18971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34133c04c7b448ef887a61783b34dab0" candidate_uuid: "326a2250ef33472f8b7c430ea003fd34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" is_pre_election: true
I20260808 04:00:45.326290 18971 raft_consensus.cc:2454] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 326a2250ef33472f8b7c430ea003fd34 in term 0.
I20260808 04:00:45.326756 19218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34133c04c7b448ef887a61783b34dab0" candidate_uuid: "326a2250ef33472f8b7c430ea003fd34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c251ae790fb4cc198421eb234585b50" is_pre_election: true
I20260808 04:00:45.326982 19333 leader_election.cc:302] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 326a2250ef33472f8b7c430ea003fd34, 84e8a49fa9e34b7884fe9d7dec8da174; no voters:
I20260808 04:00:45.327282 19218 raft_consensus.cc:2454] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 326a2250ef33472f8b7c430ea003fd34 in term 0.
I20260808 04:00:45.327385 19570 raft_consensus.cc:2781] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:00:45.327551 19570 raft_consensus.cc:482] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:00:45.327654 19570 raft_consensus.cc:3037] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:45.330025 19570 raft_consensus.cc:504] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33729 } } peers { permanent_uuid: "326a2250ef33472f8b7c430ea003fd34" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43923 } } peers { permanent_uuid: "1c251ae790fb4cc198421eb234585b50" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37167 } }
I20260808 04:00:45.330636 19570 leader_election.cc:288] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [CANDIDATE]: Term 1 election: Requested vote from peers 84e8a49fa9e34b7884fe9d7dec8da174 (127.28.251.193:33729), 1c251ae790fb4cc198421eb234585b50 (127.28.251.194:37167)
I20260808 04:00:45.331277 19218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34133c04c7b448ef887a61783b34dab0" candidate_uuid: "326a2250ef33472f8b7c430ea003fd34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c251ae790fb4cc198421eb234585b50"
I20260808 04:00:45.331277 18971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34133c04c7b448ef887a61783b34dab0" candidate_uuid: "326a2250ef33472f8b7c430ea003fd34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e8a49fa9e34b7884fe9d7dec8da174"
I20260808 04:00:45.331540 19218 raft_consensus.cc:3037] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:45.331540 18971 raft_consensus.cc:3037] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:45.334787 18971 raft_consensus.cc:2454] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 326a2250ef33472f8b7c430ea003fd34 in term 1.
I20260808 04:00:45.334789 19218 raft_consensus.cc:2454] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 326a2250ef33472f8b7c430ea003fd34 in term 1.
I20260808 04:00:45.335242 19333 leader_election.cc:302] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c251ae790fb4cc198421eb234585b50, 326a2250ef33472f8b7c430ea003fd34; no voters:
I20260808 04:00:45.335565 19570 raft_consensus.cc:2781] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:00:45.336051 19570 raft_consensus.cc:686] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 1 LEADER]: Becoming Leader. State: Replica: 326a2250ef33472f8b7c430ea003fd34, State: Running, Role: LEADER
I20260808 04:00:45.336591 19570 consensus_queue.cc:237] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33729 } } peers { permanent_uuid: "326a2250ef33472f8b7c430ea003fd34" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43923 } } peers { permanent_uuid: "1c251ae790fb4cc198421eb234585b50" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37167 } }
I20260808 04:00:45.341449 18719 catalog_manager.cc:5495] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 326a2250ef33472f8b7c430ea003fd34 (127.28.251.195). New cstate: current_term: 1 leader_uuid: "326a2250ef33472f8b7c430ea003fd34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "326a2250ef33472f8b7c430ea003fd34" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 43923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c251ae790fb4cc198421eb234585b50" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37167 } health_report { overall_health: UNKNOWN } } }
I20260808 04:00:45.391070 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.385417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.28.251.195:43923)
0808 04:00:45.385688 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:45.385694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:45.385781 (+ 87us) server_negotiation.cc:407] Connection header received
0808 04:00:45.386028 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:45.386033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:45.386117 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:45.386253 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:00:45.386697 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.387488 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.388898 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.389299 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.389412 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.389919 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:00:45.390466 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:00:45.390617 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:45.390719 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 04:00:45.390806 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 04:00:45.399550 18971 raft_consensus.cc:1261] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174 [term 1 FOLLOWER]: Refusing update from remote peer 326a2250ef33472f8b7c430ea003fd34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:00:45.399546 19218 raft_consensus.cc:1261] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50 [term 1 FOLLOWER]: Refusing update from remote peer 326a2250ef33472f8b7c430ea003fd34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:00:45.400574 19596 consensus_queue.cc:1035] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c251ae790fb4cc198421eb234585b50" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:00:45.401000 19570 consensus_queue.cc:1035] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84e8a49fa9e34b7884fe9d7dec8da174" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:00:45.430555 18719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:00:45.430850 18719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:00:45.434711 18719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bb7457951c040a2a6e08403acf2cf2a: Sending DeleteTablet for 3 replicas of tablet 34133c04c7b448ef887a61783b34dab0
I20260808 04:00:45.436053 19415 tablet_service.cc:1514] Processing DeleteTablet for tablet 34133c04c7b448ef887a61783b34dab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:00:45 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:49682
I20260808 04:00:45.436050 18940 tablet_service.cc:1514] Processing DeleteTablet for tablet 34133c04c7b448ef887a61783b34dab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:00:45 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:41904
I20260808 04:00:45.436306 19179 tablet_service.cc:1514] Processing DeleteTablet for tablet 34133c04c7b448ef887a61783b34dab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:00:45 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:48696
I20260808 04:00:45.439666 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 18784
I20260808 04:00:45.449249 19613 tablet_replica.cc:330] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: stopping tablet replica
I20260808 04:00:45.450024 19614 tablet_replica.cc:330] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50: stopping tablet replica
I20260808 04:00:45.450420 19613 raft_consensus.cc:2229] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:00:45.450623 19614 raft_consensus.cc:2229] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:00:45.451253 19614 raft_consensus.cc:2258] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:00:45.451248 19613 raft_consensus.cc:2258] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:00:45.453675 19614 ts_tablet_manager.cc:1905] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:00:45.453682 19613 ts_tablet_manager.cc:1905] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:00:45.461609 19613 ts_tablet_manager.cc:1918] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:00:45.461872 19613 log.cc:1196] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/34133c04c7b448ef887a61783b34dab0
I20260808 04:00:45.462491 19613 ts_tablet_manager.cc:1939] T 34133c04c7b448ef887a61783b34dab0 P 326a2250ef33472f8b7c430ea003fd34: Deleting consensus metadata
I20260808 04:00:45.464526 18696 catalog_manager.cc:4841] TS 326a2250ef33472f8b7c430ea003fd34 (127.28.251.195:43923): tablet 34133c04c7b448ef887a61783b34dab0 (table test-table [id=6f86207fc96b4d6091968745e7dc8a2c]) successfully deleted
I20260808 04:00:45.472138 19614 ts_tablet_manager.cc:1918] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:00:45.472452 19614 log.cc:1196] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/34133c04c7b448ef887a61783b34dab0
I20260808 04:00:45.473058 19614 ts_tablet_manager.cc:1939] T 34133c04c7b448ef887a61783b34dab0 P 1c251ae790fb4cc198421eb234585b50: Deleting consensus metadata
I20260808 04:00:45.474959 19612 tablet_replica.cc:330] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174: stopping tablet replica
I20260808 04:00:45.475046 18696 catalog_manager.cc:4841] TS 1c251ae790fb4cc198421eb234585b50 (127.28.251.194:37167): tablet 34133c04c7b448ef887a61783b34dab0 (table test-table [id=6f86207fc96b4d6091968745e7dc8a2c]) successfully deleted
I20260808 04:00:45.476528 19612 raft_consensus.cc:2229] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:00:45.477572 19612 raft_consensus.cc:2258] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:00:45.480546 19612 ts_tablet_manager.cc:1905] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:00:45.486238 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.440539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.28.251.193:33729)
0808 04:00:45.440765 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:45.440772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:45.464901 (+ 24129us) server_negotiation.cc:407] Connection header received
0808 04:00:45.464980 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:45.464990 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:45.465079 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:45.465187 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:45.467054 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.468274 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.470061 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.470786 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.470935 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.476138 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:45.476157 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:45.476177 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:45.476219 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:45.480305 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:45.483752 (+ 3447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:45.483761 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:45.483763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:45.483933 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:45.484466 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:45.484471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:45.484473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:45.484910 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:45.485249 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:45.485624 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 04:00:45.485880 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:00:45.486974 19040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:45.577708 19612 ts_tablet_manager.cc:1918] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:00:45.578012 19612 log.cc:1196] T 34133c04c7b448ef887a61783b34dab0 P 84e8a49fa9e34b7884fe9d7dec8da174: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/34133c04c7b448ef887a61783b34dab0
I20260808 04:00:45.578329 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 18784
W20260808 04:00:45.611083 18696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.251.193:33729 (error 108)
I20260808 04:00:45.611367 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 19054
I20260808 04:00:45.613821 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.612828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33729 (local address 127.0.0.1:41914)
0808 04:00:45.613522 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:45.613637 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":490,"threads_started":1}
W20260808 04:00:45.614085 18696 catalog_manager.cc:4584] TS 84e8a49fa9e34b7884fe9d7dec8da174 (127.28.251.193:33729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34133c04c7b448ef887a61783b34dab0: Network error: Client connection negotiation failed: client connection to 127.28.251.193:33729: connect: Connection refused (error 111)
I20260808 04:00:45.627663 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.612524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48712 (local address 127.28.251.194:37167)
0808 04:00:45.612715 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:45.612721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:45.612743 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:00:45.612820 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:45.612825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:45.612921 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:45.613056 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:45.614875 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.615695 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.617627 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.618035 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.618129 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.621566 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:45.621583 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:45.621599 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:45.621636 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:45.624605 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:45.625525 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:45.625536 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:45.625538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:45.625684 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:45.626097 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:45.626101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:45.626103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:45.626506 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:45.626797 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:45.627099 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 04:00:45.627295 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:00:45.628398 19280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:45.667937 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.667301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33729 (local address 127.0.0.1:41918)
0808 04:00:45.667609 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:45.667743 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260808 04:00:45.669312 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.668798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33729 (local address 127.0.0.1:41930)
0808 04:00:45.669005 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:45.669104 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 04:00:45.721032 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 19054
I20260808 04:00:45.734736 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.734172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33729 (local address 127.0.0.1:41938)
0808 04:00:45.734455 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:45.734566 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 04:00:45.736101 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.735650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33729 (local address 127.0.0.1:41940)
0808 04:00:45.735837 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:45.735928 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 04:00:45.754168 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 19295
I20260808 04:00:45.769696 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.754944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.28.251.195:43923)
0808 04:00:45.755136 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:45.755143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:45.755296 (+ 153us) server_negotiation.cc:407] Connection header received
0808 04:00:45.755530 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:45.755534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:45.755608 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:45.755709 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:45.757154 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.758019 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.759376 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:45.760042 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:45.760169 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:45.762920 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:45.762941 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:45.762959 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:45.763002 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:45.766368 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:45.767258 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:45.767267 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:45.767269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:45.767531 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:45.767938 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:45.767943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:45.767945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:45.768349 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:45.768674 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:45.769060 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 04:00:45.769294 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:00:45.770514 19521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:45.817181 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.816682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33729 (local address 127.0.0.1:41946)
0808 04:00:45.816929 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:45.817027 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 04:00:45.818321 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:45.817877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33729 (local address 127.0.0.1:41948)
0808 04:00:45.818096 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:45.818172 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260808 04:00:45.899629 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 19295
I20260808 04:00:45.931288 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 18656
I20260808 04:00:45.932626 18775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:46.049854 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:46.049262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33729 (local address 127.0.0.1:41954)
0808 04:00:46.049489 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:46.049638 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260808 04:00:46.050066 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 18656
I20260808 04:00:46.082329 29679 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 04:00:46.430663 29679 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 04:00:46.430871 29679 mini_ranger.cc:66] Stopping Ranger...
I20260808 04:00:46.788305 29679 mini_ranger.cc:68] Stopped Ranger
2026-08-08 04:00:46.788 UTC [17078] LOG: received smart shutdown request
2026-08-08 04:00:46.791 UTC [17078] LOG: background worker "logical replication launcher" (PID 17092) exited with exit code 1
2026-08-08 04:00:46.791 UTC [17083] LOG: shutting down
2026-08-08 04:00:46.949 UTC [17078] LOG: database system is shut down
2026-08-08T04:00:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57246 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:00:47 dist-test-slave-pww0 krb5kdc[19639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:00:47 dist-test-slave-pww0 krb5kdc[19639](info): set up 2 sockets
Aug 08 04:00:47 dist-test-slave-pww0 krb5kdc[19639](info): commencing operation
krb5kdc: starting...
W20260808 04:00:48.346133 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.004s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:00:48 dist-test-slave-pww0 krb5kdc[19639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:00:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:00:48.673489 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:45213
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:45213,127.28.251.253:40825,127.28.251.252:33321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:00:49.033241 19659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:49.033599 19659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:49.033741 19659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:49.041306 19659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:00:49.041460 19659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:49.041540 19659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:49.041617 19659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:00:49.041678 19659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:00:49.052238 19659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:45213,127.28.251.253:40825,127.28.251.252:33321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:45213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:49.053752 19659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:49.055171 19659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:49.067539 19665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:49.067598 19670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:49.067587 19666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:49.068471 19659 server_base.cc:1034] running on GCE node
I20260808 04:00:49.069317 19659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:49.070832 19659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:49.072052 19659 hybrid_clock.cc:648] HybridClock initialized: now 1786161649071982 us; error 55 us; skew 500 ppm
I20260808 04:00:49.072450 19659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:49 dist-test-slave-pww0 krb5kdc[19639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161649, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:49.080294 19659 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:00:49.082190 19659 webserver.cc:460] Webserver started at http://127.28.251.254:42629/ using document root <none> and password file <none>
I20260808 04:00:49.082993 19659 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:49.083128 19659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:49.083506 19659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:00:49.086964 19659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "5684616d2ad142199dcb26e8658d9555"
format_stamp: "Formatted at 2026-08-08 04:00:49 on dist-test-slave-pww0"
server_key: "f1e113a44ce665e2a9a1158a6699fc29"
server_key_iv: "1858e313479663eb3f5b5da6b5139553"
server_key_version: "encryptionkey@0"
I20260808 04:00:49.088008 19659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "5684616d2ad142199dcb26e8658d9555"
format_stamp: "Formatted at 2026-08-08 04:00:49 on dist-test-slave-pww0"
server_key: "f1e113a44ce665e2a9a1158a6699fc29"
server_key_iv: "1858e313479663eb3f5b5da6b5139553"
server_key_version: "encryptionkey@0"
I20260808 04:00:49.093735 19659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 04:00:49.097698 19683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:49.099265 19659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:00:49.099488 19659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "5684616d2ad142199dcb26e8658d9555"
format_stamp: "Formatted at 2026-08-08 04:00:49 on dist-test-slave-pww0"
server_key: "f1e113a44ce665e2a9a1158a6699fc29"
server_key_iv: "1858e313479663eb3f5b5da6b5139553"
server_key_version: "encryptionkey@0"
I20260808 04:00:49.099712 19659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:49.129818 19659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:49.134410 19659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:49.134714 19659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:49.153892 19659 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:45213
I20260808 04:00:49.153945 19771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:45213 every 8 connection(s)
I20260808 04:00:49.155601 19659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:00:49.156142 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 19659
I20260808 04:00:49.156392 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:00:49.157310 29679 external_mini_cluster.cc:1428] Setting key dbcb398e66cc4fc8838b3fa04cb3d603
I20260808 04:00:49.159926 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:00:49.166953 19772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:00:49 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161649, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.251.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.253@KRBTEST.COM" created.
I20260808 04:00:49.192463 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.170623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.28.251.254:45213)
0808 04:00:49.171678 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:49.171698 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:49.172293 (+ 595us) server_negotiation.cc:407] Connection header received
0808 04:00:49.174551 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:49.174560 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:49.174908 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:49.175133 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:49.176453 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.178254 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.179878 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.180697 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.180922 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:49.184610 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:49.184619 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:49.184634 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:49.184668 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:49.187858 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:49.188689 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:49.188697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:49.188699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:49.188865 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:49.189354 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:49.189358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:49.189360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:49.189959 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:49.190194 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:49.190868 (+ 674us) server_negotiation.cc:299] Negotiation successful
0808 04:00:49.191312 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":723,"thread_start_us":435,"threads_started":1}
I20260808 04:00:49.192488 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.170641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:45213 (local address 127.0.0.1:36754)
0808 04:00:49.172066 (+ 1425us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.172126 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:49.174085 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:49.175206 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:49.175212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:49.175867 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:49.176236 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:49.176256 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.178454 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.178458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:49.179728 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:49.179743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.180961 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.180967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:49.181092 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:49.181813 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:49.181840 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:49.184327 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:49.188026 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:49.188040 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:49.188055 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:49.188533 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:49.189011 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:49.189015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:49.189018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:49.189204 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:49.190137 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:49.190144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:49.190391 (+ 247us) client_negotiation.cc:769] Sending connection context
0808 04:00:49.190651 (+ 260us) client_negotiation.cc:240] Negotiation successful
0808 04:00:49.191149 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"spinlock_wait_cycles":22272,"thread_start_us":890,"threads_started":1}
I20260808 04:00:49.195642 19772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:00:49.196727 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.196298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52736)
0808 04:00:49.196467 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.196583 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 04:00:49.197153 19689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
I20260808 04:00:49.199295 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.198897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52746)
0808 04:00:49.199066 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.199140 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 04:00:49.199529 19772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:40825: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab.
Entry for principal kudu/127.28.251.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab.
I20260808 04:00:49.228503 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.253:40825
--webserver_interface=127.28.251.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab
--principal=kudu/127.28.251.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:45213,127.28.251.253:40825,127.28.251.252:33321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:00:49.248947 19772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } attempt: 1
I20260808 04:00:49.250417 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.249843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52752)
0808 04:00:49.250072 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.250237 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 04:00:49.251560 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.251098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52754)
0808 04:00:49.251269 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.251354 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 04:00:49.251775 19772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:40825: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
I20260808 04:00:49.320061 19772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } attempt: 2
I20260808 04:00:49.321456 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.320835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52766)
0808 04:00:49.321135 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.321263 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 04:00:49.322523 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.322097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52782)
0808 04:00:49.322260 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.322365 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 04:00:49.322708 19772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:40825: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
I20260808 04:00:49.413990 19772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } attempt: 3
I20260808 04:00:49.415364 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.414812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52792)
0808 04:00:49.415053 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.415185 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 04:00:49.416594 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.416090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52798)
0808 04:00:49.416281 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.416397 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 04:00:49.416798 19772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:40825: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
I20260808 04:00:49.560079 19772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } attempt: 4
I20260808 04:00:49.561636 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.560890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52804)
0808 04:00:49.561296 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.561437 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260808 04:00:49.562868 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.562411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52806)
0808 04:00:49.562570 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.562694 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:00:49.563104 19772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:40825: Network error: Client connection negotiation failed: client connection to 127.28.251.253:40825: connect: Connection refused (error 111)
W20260808 04:00:49.587644 19783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:49.587967 19783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:49.588069 19783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:49.595705 19783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:00:49.595876 19783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:49.595966 19783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:49.596041 19783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:00:49.596088 19783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:00:49.606593 19783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:45213,127.28.251.253:40825,127.28.251.252:33321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab
--principal=kudu/127.28.251.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.253:40825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.251.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:49.608034 19783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:49.609391 19783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:49.621968 19789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:49.622015 19790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:49.621933 19792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:49.623147 19783 server_base.cc:1034] running on GCE node
I20260808 04:00:49.624042 19783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:49.625559 19783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:49.626777 19783 hybrid_clock.cc:648] HybridClock initialized: now 1786161649626720 us; error 43 us; skew 500 ppm
I20260808 04:00:49.627147 19783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:49 dist-test-slave-pww0 krb5kdc[19639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161649, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:49.630478 19783 init.cc:377] Logged in from keytab as kudu/127.28.251.253@KRBTEST.COM (short username kudu)
I20260808 04:00:49.632223 19783 webserver.cc:460] Webserver started at http://127.28.251.253:42547/ using document root <none> and password file <none>
I20260808 04:00:49.632965 19783 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:49.633078 19783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:49.633365 19783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:00:49.636746 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/data/instance:
uuid: "4c4f36a3814f4ee991c107c78a1f9e25"
format_stamp: "Formatted at 2026-08-08 04:00:49 on dist-test-slave-pww0"
server_key: "66a3b515e42e61f8279455a052eae39b"
server_key_iv: "da0d9969a1aced77ef4621f86d014354"
server_key_version: "encryptionkey@0"
I20260808 04:00:49.637682 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/wal/instance:
uuid: "4c4f36a3814f4ee991c107c78a1f9e25"
format_stamp: "Formatted at 2026-08-08 04:00:49 on dist-test-slave-pww0"
server_key: "66a3b515e42e61f8279455a052eae39b"
server_key_iv: "da0d9969a1aced77ef4621f86d014354"
server_key_version: "encryptionkey@0"
I20260808 04:00:49.643672 19783 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 04:00:49.647575 19801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:49.649102 19783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:00:49.649304 19783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/wal
uuid: "4c4f36a3814f4ee991c107c78a1f9e25"
format_stamp: "Formatted at 2026-08-08 04:00:49 on dist-test-slave-pww0"
server_key: "66a3b515e42e61f8279455a052eae39b"
server_key_iv: "da0d9969a1aced77ef4621f86d014354"
server_key_version: "encryptionkey@0"
I20260808 04:00:49.649518 19783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:49.676502 19783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:49.680751 19783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:49.681042 19783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:49.699697 19783 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.253:40825
I20260808 04:00:49.699733 19886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.253:40825 every 8 connection(s)
I20260808 04:00:49.701116 19783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
I20260808 04:00:49.701736 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 19783
I20260808 04:00:49.701923 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-1/wal/instance
I20260808 04:00:49.702530 29679 external_mini_cluster.cc:1428] Setting key 4c899f3fce044bd20dbe7f8a78c0c9b1
I20260808 04:00:49.705462 19890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:00:49.712446 19890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:00:49 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161649, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.251.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.252@KRBTEST.COM" created.
I20260808 04:00:49.735673 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.715209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.28.251.254:45213)
0808 04:00:49.715771 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:49.715776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:49.716999 (+ 1223us) server_negotiation.cc:407] Connection header received
0808 04:00:49.718965 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:49.718971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:49.719057 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:49.719182 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:49.720860 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.721996 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.723732 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.724303 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.724434 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:49.728464 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:49.728471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:49.728476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:49.728538 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:49.732072 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:49.732991 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:49.732996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:49.732999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:49.733132 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:49.733597 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:49.733601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:49.733603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:49.733942 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:49.734071 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:49.734989 (+ 918us) server_negotiation.cc:299] Negotiation successful
0808 04:00:49.735234 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":314,"threads_started":1}
I20260808 04:00:49.736894 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.715946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:45213 (local address 127.0.0.1:36768)
0808 04:00:49.716799 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.716856 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:49.718675 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:49.719402 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:49.719411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:49.720014 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:49.720631 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:49.720649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.722219 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.722224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:49.723580 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:49.723594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.724577 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.724583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:49.724817 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:49.725556 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:49.725581 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:49.728116 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:49.732258 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:49.732267 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:49.732283 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:49.732845 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:49.733219 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:49.733223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:49.733225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:49.733439 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:49.734063 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:49.734070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:49.734463 (+ 393us) client_negotiation.cc:769] Sending connection context
0808 04:00:49.734820 (+ 357us) client_negotiation.cc:240] Negotiation successful
0808 04:00:49.735376 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":345,"threads_started":1}
I20260808 04:00:49.739027 19890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:00:49 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161649, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
I20260808 04:00:49.755759 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.739761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52814)
0808 04:00:49.739948 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.739970 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:49.740092 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:49.741298 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:49.741301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:49.741704 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:49.741855 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:49.741864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.743726 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.743731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:49.744596 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:49.744608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.745742 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.745748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:49.745896 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:49.746610 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:49.746631 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:49.749017 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:49.752627 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:49.752633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:49.752637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:49.753198 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:49.753702 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:49.753707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:49.753711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:49.753909 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:49.754705 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:49.754711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:49.754874 (+ 163us) client_negotiation.cc:769] Sending connection context
0808 04:00:49.755032 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 04:00:49.755357 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 04:00:49.755870 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.740052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.28.251.253:40825)
0808 04:00:49.740716 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:49.740725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:49.740759 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:00:49.740823 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:49.740831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:49.741112 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:49.741282 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:49.742021 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.743545 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.744748 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.745573 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.745697 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:49.749285 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:49.749305 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:49.749326 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:49.749386 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:49.752452 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:49.753368 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:49.753377 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:49.753379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:49.753570 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:49.754051 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:49.754057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:49.754061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:49.754562 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:49.754765 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:49.755163 (+ 398us) server_negotiation.cc:299] Negotiation successful
0808 04:00:49.755539 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":464,"thread_start_us":298,"threads_started":1}
I20260808 04:00:49.758577 19890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:00:49.760057 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.759405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:39964)
0808 04:00:49.759629 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.759861 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260808 04:00:49.760422 19805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
I20260808 04:00:49.762157 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.761703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:39976)
0808 04:00:49.761906 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.762009 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 04:00:49.762404 19890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:33321: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab.
Entry for principal kudu/127.28.251.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab.
I20260808 04:00:49.774927 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.252:33321
--webserver_interface=127.28.251.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab
--principal=kudu/127.28.251.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:45213,127.28.251.253:40825,127.28.251.252:33321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:00:49.811775 19890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } attempt: 1
I20260808 04:00:49.813124 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.812544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:39992)
0808 04:00:49.812790 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.812925 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260808 04:00:49.814196 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.813806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:39994)
0808 04:00:49.813962 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.814054 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 04:00:49.814415 19890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:33321: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
I20260808 04:00:49.862414 19772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } attempt: 5
Aug 08 04:00:49 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161649, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
I20260808 04:00:49.877509 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.863382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.28.251.253:40825)
0808 04:00:49.863595 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:49.863600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:49.863620 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:00:49.863797 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:49.863802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:49.863887 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:49.863970 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:49.865008 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.865703 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.867434 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.867958 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.868060 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:49.872106 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:49.872111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:49.872115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:49.872155 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:49.874751 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:49.875633 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:49.875637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:49.875639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:49.875768 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:49.876176 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:49.876179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:49.876181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:49.876482 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:49.876601 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:49.877078 (+ 477us) server_negotiation.cc:299] Negotiation successful
0808 04:00:49.877268 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:00:49.877665 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.863239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52826)
0808 04:00:49.863498 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.863521 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:49.863661 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:49.864045 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:49.864051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:49.864557 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:49.864826 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:49.864839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.865853 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.865859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:49.867258 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:49.867277 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:49.868119 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:49.868126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:49.868322 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:49.869044 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:49.869068 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:49.871861 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:49.874931 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:49.874937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:49.874941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:49.875520 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:49.875877 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:49.875893 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:49.875897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:49.876078 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:49.876609 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:49.876615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:49.876823 (+ 208us) client_negotiation.cc:769] Sending connection context
0808 04:00:49.876981 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 04:00:49.877327 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 04:00:49.878796 19772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:00:49.880175 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.879682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40000)
0808 04:00:49.879888 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.879996 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 04:00:49.881247 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.880867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40002)
0808 04:00:49.881014 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.881109 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:00:49.881498 19772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:33321: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
I20260808 04:00:49.882624 19890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } attempt: 2
I20260808 04:00:49.883641 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.883151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40006)
0808 04:00:49.883330 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.883429 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:00:49.884579 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.884208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40020)
0808 04:00:49.884349 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.884435 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 04:00:49.884841 19890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:33321: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
I20260808 04:00:49.932821 19772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } attempt: 1
I20260808 04:00:49.934347 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.933704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40028)
0808 04:00:49.933984 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.934112 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260808 04:00:49.935791 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.935247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40030)
0808 04:00:49.935435 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.935604 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 04:00:49.936095 19772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:33321: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
I20260808 04:00:49.976078 19890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } attempt: 3
I20260808 04:00:49.977454 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.976839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40038)
0808 04:00:49.977105 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.977238 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 04:00:49.978719 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:49.978158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40046)
0808 04:00:49.978429 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:49.978542 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260808 04:00:49.978951 19890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:33321: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
I20260808 04:00:50.004346 19772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } attempt: 2
I20260808 04:00:50.005721 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.005099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40062)
0808 04:00:50.005372 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:50.005496 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 04:00:50.006896 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.006476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40074)
0808 04:00:50.006652 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:50.006746 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:00:50.007149 19772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:33321: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
I20260808 04:00:50.113417 19772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } attempt: 3
I20260808 04:00:50.114753 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.114182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40086)
0808 04:00:50.114425 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:50.114543 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 04:00:50.116299 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.115765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40090)
0808 04:00:50.116023 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:50.116126 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260808 04:00:50.116542 19772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:33321: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
I20260808 04:00:50.122218 19890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } attempt: 4
I20260808 04:00:50.123534 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.122868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40092)
0808 04:00:50.123149 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:50.123282 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 04:00:50.124701 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.124290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40094)
0808 04:00:50.124472 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:50.124565 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:00:50.124951 19890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:33321: Network error: Client connection negotiation failed: client connection to 127.28.251.252:33321: connect: Connection refused (error 111)
W20260808 04:00:50.148680 19899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:50.149001 19899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:50.149103 19899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:50.156870 19899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:00:50.156993 19899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:50.157058 19899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:50.157141 19899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:00:50.157203 19899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:00:50.167910 19899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:45213,127.28.251.253:40825,127.28.251.252:33321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab
--principal=kudu/127.28.251.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.252:33321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.251.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:50.169410 19899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:50.170877 19899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:50.183347 19907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:50.183413 19913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:50.183445 19909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:50.184581 19899 server_base.cc:1034] running on GCE node
I20260808 04:00:50.185490 19899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:50.187131 19899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:50.188375 19899 hybrid_clock.cc:648] HybridClock initialized: now 1786161650188298 us; error 71 us; skew 500 ppm
I20260808 04:00:50.188777 19899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:50 dist-test-slave-pww0 krb5kdc[19639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161650, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:50.192229 19899 init.cc:377] Logged in from keytab as kudu/127.28.251.252@KRBTEST.COM (short username kudu)
I20260808 04:00:50.193938 19899 webserver.cc:460] Webserver started at http://127.28.251.252:38989/ using document root <none> and password file <none>
I20260808 04:00:50.194720 19899 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:50.194849 19899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:50.195179 19899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:00:50.198561 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/data/instance:
uuid: "818c95e02f524c7591279af76aafe3ae"
format_stamp: "Formatted at 2026-08-08 04:00:50 on dist-test-slave-pww0"
server_key: "3b9d64564b6a89da5c4379fdf4045473"
server_key_iv: "2a2d00ceee209c881c7529ff384f0586"
server_key_version: "encryptionkey@0"
I20260808 04:00:50.199435 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/wal/instance:
uuid: "818c95e02f524c7591279af76aafe3ae"
format_stamp: "Formatted at 2026-08-08 04:00:50 on dist-test-slave-pww0"
server_key: "3b9d64564b6a89da5c4379fdf4045473"
server_key_iv: "2a2d00ceee209c881c7529ff384f0586"
server_key_version: "encryptionkey@0"
I20260808 04:00:50.205498 19899 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 04:00:50.209614 19922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:50.211190 19899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:00:50.211396 19899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/wal
uuid: "818c95e02f524c7591279af76aafe3ae"
format_stamp: "Formatted at 2026-08-08 04:00:50 on dist-test-slave-pww0"
server_key: "3b9d64564b6a89da5c4379fdf4045473"
server_key_iv: "2a2d00ceee209c881c7529ff384f0586"
server_key_version: "encryptionkey@0"
I20260808 04:00:50.211676 19899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:50.245791 19899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:50.250419 19899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:50.250735 19899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:50.269891 19899 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.252:33321
I20260808 04:00:50.269932 20008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.252:33321 every 8 connection(s)
I20260808 04:00:50.271375 19899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
I20260808 04:00:50.275368 20009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:00:50.279146 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 19899
I20260808 04:00:50.279368 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786161430375636-29679-0/minicluster-data/master-2/wal/instance
I20260808 04:00:50.280025 29679 external_mini_cluster.cc:1428] Setting key 11b74e7c6140a3f0766953d7de2e7e59
I20260808 04:00:50.283726 20009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:00:50.293771 19772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } attempt: 4
Aug 08 04:00:50 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
Aug 08 04:00:50 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161650, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
Aug 08 04:00:50 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161649, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
I20260808 04:00:50.305452 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.287377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.28.251.254:45213)
0808 04:00:50.288723 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:50.288729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:50.288749 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:00:50.288809 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:50.288814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:50.288888 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:50.288989 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:50.290143 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.291139 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.292981 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.293578 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.293762 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.298111 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:50.298115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:50.298118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:50.298151 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:50.301518 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.302558 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.302565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.302568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.302748 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.303228 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.303234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.303238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.303597 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:50.303744 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:50.304938 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0808 04:00:50.305152 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1226,"thread_start_us":1045,"threads_started":1}
I20260808 04:00:50.306444 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.282288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.28.251.252:33321)
0808 04:00:50.283138 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:50.283151 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:50.283198 (+ 47us) server_negotiation.cc:407] Connection header received
0808 04:00:50.284904 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:50.284930 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:50.285365 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:50.285742 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:50.287359 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.289332 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.291042 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.292077 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.292283 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.297739 (+ 5456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:50.297755 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:50.297775 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:50.297826 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:50.301192 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.302097 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.302105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.302107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.302354 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.302841 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.302846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.302849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.303664 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:50.303996 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:50.304459 (+ 463us) server_negotiation.cc:299] Negotiation successful
0808 04:00:50.305002 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":631,"spinlock_wait_cycles":52480,"thread_start_us":404,"threads_started":1}
I20260808 04:00:50.306466 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.287270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:45213 (local address 127.0.0.1:36780)
0808 04:00:50.288014 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:50.288058 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:50.288220 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:50.289067 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:50.289072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:50.289698 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:50.289976 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:50.289986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.291307 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.291312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:50.292827 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:50.292842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.293830 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.293839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:50.294016 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.295178 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:50.295205 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:50.297906 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:50.301827 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:50.301841 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:50.301854 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:50.302436 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:50.302887 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:50.302891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:50.302893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:50.303108 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:50.303831 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:50.303851 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:50.304165 (+ 314us) client_negotiation.cc:769] Sending connection context
0808 04:00:50.304458 (+ 293us) client_negotiation.cc:240] Negotiation successful
0808 04:00:50.305898 (+ 1440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":897,"thread_start_us":321,"threads_started":1}
I20260808 04:00:50.309487 20009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:00:50.312181 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.294430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40108)
0808 04:00:50.294591 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:50.294610 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:50.294697 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:50.295662 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:50.295666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:50.296126 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:50.296328 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:50.296340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.299120 (+ 2780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.299126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:50.300745 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:50.300762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.301692 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.301699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:50.301906 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.302473 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:50.302508 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:50.305344 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:50.309301 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:50.309307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:50.309310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:50.309904 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:50.310362 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:50.310368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:50.310371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:50.310559 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:50.311098 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:50.311103 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:50.311276 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 04:00:50.311430 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 04:00:50.311782 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 04:00:50.312484 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.294582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.28.251.252:33321)
0808 04:00:50.295302 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:50.295308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:50.295331 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:00:50.295377 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:50.295382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:50.295488 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:50.295606 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:50.297587 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.298969 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.300927 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.301535 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.301640 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.305600 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:50.305607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:50.305612 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:50.305654 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:50.309151 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.310032 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.310037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.310041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.310204 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.310675 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.310679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.310681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.310977 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:50.311110 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:50.311551 (+ 441us) server_negotiation.cc:299] Negotiation successful
0808 04:00:50.311794 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":351,"threads_started":1}
Aug 08 04:00:50 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161650, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
I20260808 04:00:50.318959 19772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555: Bootstrap starting.
Aug 08 04:00:50 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:50.322973 19772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:50.324503 19772 log.cc:824] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:50.325258 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.310275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.28.251.253:40825)
0808 04:00:50.310607 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:50.310612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:50.311268 (+ 656us) server_negotiation.cc:407] Connection header received
0808 04:00:50.311394 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:50.311399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:50.311514 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:50.311603 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:50.312650 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.313349 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.314841 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.315575 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.315724 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.319039 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:50.319045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:50.319049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:50.319086 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:50.322358 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.323099 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.323104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.323108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.323287 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.323758 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.323763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.323767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.324050 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:50.324202 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:50.324656 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 04:00:50.324872 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260808 04:00:50.325273 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.310487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:40825 (local address 127.0.0.1:52840)
0808 04:00:50.311163 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:50.311180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:50.311289 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:50.311804 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:50.311809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:50.312294 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:50.312498 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:50.312508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.313491 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.313496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:50.314657 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:50.314684 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.315727 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.315734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:50.315992 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.316613 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:50.316638 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:50.318829 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:50.322509 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:50.322515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:50.322519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:50.322963 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:50.323431 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:50.323435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:50.323438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:50.323632 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:50.324185 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:50.324190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:50.324373 (+ 183us) client_negotiation.cc:769] Sending connection context
0808 04:00:50.324533 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 04:00:50.324855 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562}
I20260808 04:00:50.326628 20009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:00:50.330546 19772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555: No bootstrap required, opened a new log
I20260808 04:00:50.331966 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.310413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.28.251.254:45213)
0808 04:00:50.310727 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:50.310731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:50.310745 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:00:50.310782 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:50.310786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:50.310860 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:50.310945 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:50.312672 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.314004 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.315596 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.316646 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.316793 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.322333 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:50.322349 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:50.322354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:50.322398 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:50.328225 (+ 5827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.329102 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.329109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.329112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.329317 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.329897 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.329903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.329906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.330332 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:50.330689 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:50.331356 (+ 667us) server_negotiation.cc:299] Negotiation successful
0808 04:00:50.331647 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
Aug 08 04:00:50 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161650, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
I20260808 04:00:50.340976 19772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } }
I20260808 04:00:50.341394 19772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:50.341531 19772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5684616d2ad142199dcb26e8658d9555, State: Initialized, Role: FOLLOWER
I20260808 04:00:50.342247 19772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } }
I20260808 04:00:50.343731 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.327621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40110)
0808 04:00:50.327831 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:50.327854 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:50.327969 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:50.329107 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:50.329110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:50.329536 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:50.329756 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:50.329768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.331132 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.331136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:50.332378 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:50.332394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.333443 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.333447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:50.333593 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.334248 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:50.334273 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:50.336835 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:50.340548 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:50.340554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:50.340558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:50.341066 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:50.341562 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:50.341566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:50.341569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:50.341733 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:50.342493 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:50.342498 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:50.342696 (+ 198us) client_negotiation.cc:769] Sending connection context
0808 04:00:50.342868 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 04:00:50.343263 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
Aug 08 04:00:50 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
I20260808 04:00:50.344610 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.327883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.28.251.252:33321)
0808 04:00:50.328052 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:50.328058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:50.328078 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:00:50.328132 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:50.328137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:50.328224 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:50.328330 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:50.329915 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.330971 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.332533 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.333141 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.333259 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.337028 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:50.337033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:50.337037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:50.337067 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:50.340389 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.341223 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.341229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.341232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.341410 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.341871 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.341877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.341881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.342348 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:50.342671 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:50.343022 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 04:00:50.344281 (+ 1259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":970,"server-negotiator.queue_time_us":54}
I20260808 04:00:50.344652 20019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } } }
I20260808 04:00:50.345513 20019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:00:50.346778 19772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:00:50.350140 20009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae: Bootstrap starting.
I20260808 04:00:50.352211 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.334103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.28.251.253:40825)
0808 04:00:50.334310 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:50.334316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:50.334346 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:00:50.334447 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:50.334452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:50.334534 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:50.334638 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:50.336432 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.337583 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.339071 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.339587 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.339704 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.345108 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:50.345113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:50.345116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:50.345154 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:50.348675 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.349664 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.349671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.349675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.349882 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.350467 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.350473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.350476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.350883 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:50.351198 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:50.351519 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 04:00:50.351811 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 04:00:50.354604 20009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:50.356112 20009 log.cc:824] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:50.358659 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:00:50.359344 20009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae: No bootstrap required, opened a new log
I20260808 04:00:50.368949 20009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } }
I20260808 04:00:50.369467 20009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:50.369587 20009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818c95e02f524c7591279af76aafe3ae, State: Initialized, Role: FOLLOWER
I20260808 04:00:50.370515 20009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } }
I20260808 04:00:50.371178 19659 master_runner.cc:186] Error getting master registration for 127.28.251.253:40825: OK, instance_id { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" instance_seqno: 1786161649694373 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:00:50.382475 19659 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:00:50.383733 20051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } } }
I20260808 04:00:50.384268 20051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:00:50.385262 20009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:00:50.386063 20049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:00:50.386626 20049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260808 04:00:50.387929 19693 tablet.cc:2369] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:00:50.409684 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.384931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.28.251.254:45213)
0808 04:00:50.385113 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:50.385124 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:50.389314 (+ 4190us) server_negotiation.cc:407] Connection header received
0808 04:00:50.389619 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:50.389625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:50.389713 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:50.389786 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:50.392513 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.393722 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.395892 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.396724 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.396829 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.400894 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:50.400901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:50.400907 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:50.400980 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:50.405107 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.406480 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.406488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.406492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.406699 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.407193 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.407199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.407202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.407640 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:50.407936 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:50.408989 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0808 04:00:50.409265 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:00:50.413672 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.380426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.28.251.252:33321)
0808 04:00:50.380619 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:50.380626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:50.383425 (+ 2799us) server_negotiation.cc:407] Connection header received
0808 04:00:50.383795 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:50.383801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:50.383966 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:50.384058 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:50.386762 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.388101 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.390877 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.391676 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.391796 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.406082 (+ 14286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:50.406090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:50.406095 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:50.406142 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:50.410057 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.410940 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.410946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.410949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.411140 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.411686 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.411691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.411694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.412093 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:50.412403 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:50.413041 (+ 638us) server_negotiation.cc:299] Negotiation successful
0808 04:00:50.413285 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":69}
I20260808 04:00:50.417017 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.389540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.28.251.253:40825)
0808 04:00:50.389733 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:50.389739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:50.396629 (+ 6890us) server_negotiation.cc:407] Connection header received
0808 04:00:50.397104 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:50.397111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:50.397197 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:50.397267 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:50.399570 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.400769 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.402829 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.403558 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.403657 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.407903 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:50.407910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:50.407915 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:50.407960 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:50.411614 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.414425 (+ 2811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.414431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.414435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.414616 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.415131 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.415138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.415141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.415533 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:50.415806 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:50.416379 (+ 573us) server_negotiation.cc:299] Negotiation successful
0808 04:00:50.416629 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 04:00:50.424176 19890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } attempt: 5
I20260808 04:00:50.427021 20078 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Aug 08 04:00:50 dist-test-slave-pww0 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161649, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
I20260808 04:00:50.440790 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.424720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:33321 (local address 127.0.0.1:40120)
0808 04:00:50.424985 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:50.425005 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:50.425123 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:50.425433 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:50.425438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:50.425856 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:50.426018 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:50.426027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.427127 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.427133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:50.428288 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:50.428302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.429254 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.429260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:50.429495 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.430197 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:50.430223 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:50.433280 (+ 3057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:50.437624 (+ 4344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:50.437630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:50.437635 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:50.438207 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:50.438745 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:50.438751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:50.438754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:50.438957 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:50.439721 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:50.439727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:50.439900 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 04:00:50.440059 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 04:00:50.440393 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260808 04:00:50.440838 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:50.424824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.28.251.252:33321)
0808 04:00:50.424987 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:50.424992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:50.425070 (+ 78us) server_negotiation.cc:407] Connection header received
0808 04:00:50.425233 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:50.425239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:50.425308 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:50.425389 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:50.426159 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.426996 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.428434 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:50.429079 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:50.429842 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:50.433496 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:50.433503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:50.433507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:50.433550 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:50.437434 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.438393 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.438398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.438402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.438576 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:50.439117 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:50.439123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:50.439127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:50.439560 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:50.439767 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:50.440184 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 04:00:50.440451 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:00:50.445911 19890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25: Bootstrap starting.
I20260808 04:00:50.435074 19899 master_runner.cc:186] Error getting master registration for 127.28.251.253:40825: OK, instance_id { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" instance_seqno: 1786161649694373 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:00:50.447189 19899 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:00:50.449970 19890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:50.451560 19890 log.cc:824] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:50.454907 19890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25: No bootstrap required, opened a new log
I20260808 04:00:50.463567 19890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } }
I20260808 04:00:50.463936 19890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:50.464049 19890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4f36a3814f4ee991c107c78a1f9e25, State: Initialized, Role: FOLLOWER
I20260808 04:00:50.464743 19890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } }
I20260808 04:00:50.465741 20080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } } }
I20260808 04:00:50.466152 20080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:00:50.466989 19890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:00:50.480472 20096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:00:50.480741 20096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:00:50.487560 19783 master_runner.cc:263] No leader master found from master 4c4f36a3814f4ee991c107c78a1f9e25
I20260808 04:00:50.487751 19783 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 04:00:50.503216 19931 tablet.cc:2369] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:00:50.544891 20051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:00:50.545220 20051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } }
I20260808 04:00:50.547055 20051 leader_election.cc:288] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5684616d2ad142199dcb26e8658d9555 (127.28.251.254:45213), 4c4f36a3814f4ee991c107c78a1f9e25 (127.28.251.253:40825)
I20260808 04:00:50.547950 19729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "818c95e02f524c7591279af76aafe3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684616d2ad142199dcb26e8658d9555" is_pre_election: true
I20260808 04:00:50.548060 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "818c95e02f524c7591279af76aafe3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" is_pre_election: true
I20260808 04:00:50.548625 19729 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818c95e02f524c7591279af76aafe3ae in term 0.
I20260808 04:00:50.548731 19845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 818c95e02f524c7591279af76aafe3ae in term 0.
I20260808 04:00:50.549531 19928 leader_election.cc:302] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684616d2ad142199dcb26e8658d9555, 818c95e02f524c7591279af76aafe3ae; no voters:
I20260808 04:00:50.550033 20051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:00:50.550215 20051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:00:50.550454 20051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:50.553937 20051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } }
I20260808 04:00:50.554756 20051 leader_election.cc:288] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [CANDIDATE]: Term 1 election: Requested vote from peers 5684616d2ad142199dcb26e8658d9555 (127.28.251.254:45213), 4c4f36a3814f4ee991c107c78a1f9e25 (127.28.251.253:40825)
I20260808 04:00:50.555510 19729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "818c95e02f524c7591279af76aafe3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684616d2ad142199dcb26e8658d9555"
I20260808 04:00:50.555514 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "818c95e02f524c7591279af76aafe3ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c4f36a3814f4ee991c107c78a1f9e25"
I20260808 04:00:50.555794 19845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:50.555780 19729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:50.559481 19729 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818c95e02f524c7591279af76aafe3ae in term 1.
I20260808 04:00:50.559486 19845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818c95e02f524c7591279af76aafe3ae in term 1.
I20260808 04:00:50.560196 19928 leader_election.cc:302] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c4f36a3814f4ee991c107c78a1f9e25, 818c95e02f524c7591279af76aafe3ae; no voters:
I20260808 04:00:50.560637 20051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:00:50.563683 20051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [term 1 LEADER]: Becoming Leader. State: Replica: 818c95e02f524c7591279af76aafe3ae, State: Running, Role: LEADER
I20260808 04:00:50.564631 20051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } }
I20260808 04:00:50.569521 20099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 818c95e02f524c7591279af76aafe3ae. Latest consensus state: current_term: 1 leader_uuid: "818c95e02f524c7591279af76aafe3ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } } }
I20260808 04:00:50.570005 20099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [sys.catalog]: This master's current role is: LEADER
I20260808 04:00:50.574527 20102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:00:50.581429 20102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:00:50.600147 19845 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [term 1 FOLLOWER]: Refusing update from remote peer 818c95e02f524c7591279af76aafe3ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:00:50.601289 20051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:00:50.601626 19729 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [term 1 FOLLOWER]: Refusing update from remote peer 818c95e02f524c7591279af76aafe3ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:00:50.602847 20051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:00:50.623497 20080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 818c95e02f524c7591279af76aafe3ae. Latest consensus state: current_term: 1 leader_uuid: "818c95e02f524c7591279af76aafe3ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } } }
I20260808 04:00:50.623889 20080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:00:50.627979 20108 mvcc.cc:205] Tried to move back new op lower bound from 7316118120843415552 to 7316118120718192640. Current Snapshot: MvccSnapshot[applied={T|T < 7316118120843415552}]
I20260808 04:00:50.632949 20109 mvcc.cc:205] Tried to move back new op lower bound from 7316118120843415552 to 7316118120718192640. Current Snapshot: MvccSnapshot[applied={T|T < 7316118120843415552}]
I20260808 04:00:50.633919 20080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "818c95e02f524c7591279af76aafe3ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } } }
I20260808 04:00:50.634191 20080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:00:50.640198 20051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "818c95e02f524c7591279af76aafe3ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } } }
I20260808 04:00:50.640628 20051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [sys.catalog]: This master's current role is: LEADER
I20260808 04:00:50.642849 20102 catalog_manager.cc:1347] Generated new cluster ID: 7fdcfe039b444202b13b3603ef3fbc19
I20260808 04:00:50.642977 20102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:00:50.646600 20019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 818c95e02f524c7591279af76aafe3ae. Latest consensus state: current_term: 1 leader_uuid: "818c95e02f524c7591279af76aafe3ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } } }
I20260808 04:00:50.646919 20019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:00:50.650682 20099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "818c95e02f524c7591279af76aafe3ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } } }
I20260808 04:00:50.651002 20099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae [sys.catalog]: This master's current role is: LEADER
I20260808 04:00:50.664155 20019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "818c95e02f524c7591279af76aafe3ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5684616d2ad142199dcb26e8658d9555" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45213 } } peers { permanent_uuid: "4c4f36a3814f4ee991c107c78a1f9e25" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 40825 } } peers { permanent_uuid: "818c95e02f524c7591279af76aafe3ae" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 33321 } } }
I20260808 04:00:50.664489 20019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:00:50.685678 20102 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:00:50.687934 20102 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:00:50.708424 20102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae: Generated new TSK 0
I20260808 04:00:50.709645 20102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:00:51.388880 20049 catalog_manager.cc:1259] Loaded cluster ID: 7fdcfe039b444202b13b3603ef3fbc19
I20260808 04:00:51.388986 20049 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555: loading cluster ID for follower catalog manager: success
I20260808 04:00:51.391799 20049 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555: acquiring CA information for follower catalog manager: success
I20260808 04:00:51.393764 20049 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5684616d2ad142199dcb26e8658d9555: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:00:51.483098 20096 catalog_manager.cc:1259] Loaded cluster ID: 7fdcfe039b444202b13b3603ef3fbc19
I20260808 04:00:51.483217 20096 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25: loading cluster ID for follower catalog manager: success
I20260808 04:00:51.485889 20096 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25: acquiring CA information for follower catalog manager: success
I20260808 04:00:51.487979 20096 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c4f36a3814f4ee991c107c78a1f9e25: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:00:51.843025 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 19659
I20260808 04:00:51.844568 19767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:51.958164 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 19659
I20260808 04:00:51.987859 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 19783
I20260808 04:00:51.989104 19878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:52.099678 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 19783
I20260808 04:00:52.120640 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:52.119780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:45213 (local address 127.0.0.1:49214)
0808 04:00:52.120293 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:52.120456 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":281,"threads_started":1}
W20260808 04:00:52.121014 19928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.251.254:45213: connect: Connection refused (error 111)
I20260808 04:00:52.123024 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:52.122563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:45213 (local address 127.0.0.1:49230)
0808 04:00:52.122747 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:52.122866 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 04:00:52.123255 19928 consensus_peers.cc:487] T 00000000000000000000000000000000 P 818c95e02f524c7591279af76aafe3ae -> Peer 5684616d2ad142199dcb26e8658d9555 (127.28.251.254:45213): Couldn't send request to peer 5684616d2ad142199dcb26e8658d9555. Status: Network error: Client connection negotiation failed: client connection to 127.28.251.254:45213: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 04:00:52.130810 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 19899
I20260808 04:00:52.132092 20001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:52.244840 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 19899
2026-08-08T04:00:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5194 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:00:52 dist-test-slave-pww0 krb5kdc[20134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:00:52 dist-test-slave-pww0 krb5kdc[20134](info): set up 2 sockets
Aug 08 04:00:52 dist-test-slave-pww0 krb5kdc[20134](info): commencing operation
krb5kdc: starting...
W20260808 04:00:53.544546 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:00:53 dist-test-slave-pww0 krb5kdc[20134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:00:53.878894 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:33213
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:33213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:00:54.243772 20152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:54.244123 20152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:54.244266 20152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:54.251823 20152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:00:54.251964 20152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:54.252048 20152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:54.252125 20152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:00:54.252185 20152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:00:54.262485 20152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:33213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:33213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:54.264043 20152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:54.265405 20152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:54.277719 20162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:54.278053 20161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:54.277722 20166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:54.278728 20152 server_base.cc:1034] running on GCE node
I20260808 04:00:54.279657 20152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:54.281330 20152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:54.282634 20152 hybrid_clock.cc:648] HybridClock initialized: now 1786161654282545 us; error 33 us; skew 500 ppm
I20260808 04:00:54.283043 20152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:54 dist-test-slave-pww0 krb5kdc[20134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161654, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:54.289458 20152 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:00:54.291193 20152 webserver.cc:460] Webserver started at http://127.28.251.254:33061/ using document root <none> and password file <none>
I20260808 04:00:54.292011 20152 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:54.292129 20152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:54.292452 20152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:00:54.295691 20152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "d2fec72677884217a81035f48bd5afc4"
format_stamp: "Formatted at 2026-08-08 04:00:54 on dist-test-slave-pww0"
server_key: "4154c5902de417082121e55bfac9c354"
server_key_iv: "38cd1afcc9900d00a64fdb21f0d6e3c7"
server_key_version: "encryptionkey@0"
I20260808 04:00:54.296499 20152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "d2fec72677884217a81035f48bd5afc4"
format_stamp: "Formatted at 2026-08-08 04:00:54 on dist-test-slave-pww0"
server_key: "4154c5902de417082121e55bfac9c354"
server_key_iv: "38cd1afcc9900d00a64fdb21f0d6e3c7"
server_key_version: "encryptionkey@0"
I20260808 04:00:54.302106 20152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:00:54.305868 20185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:54.307271 20152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:00:54.307482 20152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "d2fec72677884217a81035f48bd5afc4"
format_stamp: "Formatted at 2026-08-08 04:00:54 on dist-test-slave-pww0"
server_key: "4154c5902de417082121e55bfac9c354"
server_key_iv: "38cd1afcc9900d00a64fdb21f0d6e3c7"
server_key_version: "encryptionkey@0"
I20260808 04:00:54.307708 20152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:54.323520 20152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:54.327901 20152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:54.328189 20152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:54.346751 20152 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:33213
I20260808 04:00:54.346789 20259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:33213 every 8 connection(s)
I20260808 04:00:54.348245 20152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:00:54.352193 20261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:00:54.353048 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 20152
I20260808 04:00:54.353304 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:00:54.354229 29679 external_mini_cluster.cc:1428] Setting key 6b7eefba07ce3d220b0bcf71d0e3e97e
I20260808 04:00:54.367300 20261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4: Bootstrap starting.
Aug 08 04:00:54 dist-test-slave-pww0 krb5kdc[20134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:54.371515 20261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:54.372812 20261 log.cc:824] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:54.376567 20261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4: No bootstrap required, opened a new log
I20260808 04:00:54.379832 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:54.356310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.28.251.254:33213)
0808 04:00:54.357142 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:54.357154 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:54.357203 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:00:54.358940 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:54.358969 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:54.359409 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:54.359820 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:54.361859 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:54.363265 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:54.365179 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:54.365922 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:54.366057 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:54.371197 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:54.371206 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:54.371223 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:54.371267 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:54.374859 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:54.375939 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:54.375946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:54.375955 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:54.376140 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:54.376624 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:54.376627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:54.376629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:54.377285 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:54.377543 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:54.378070 (+ 527us) server_negotiation.cc:299] Negotiation successful
0808 04:00:54.378467 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":344,"threads_started":1}
I20260808 04:00:54.384336 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:00:54.385917 20261 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fec72677884217a81035f48bd5afc4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33213 } }
I20260808 04:00:54.386344 20261 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:54.386446 20261 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fec72677884217a81035f48bd5afc4, State: Initialized, Role: FOLLOWER
I20260808 04:00:54.387137 20261 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fec72677884217a81035f48bd5afc4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33213 } }
I20260808 04:00:54.387418 20261 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:00:54.387609 20261 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:00:54.387789 20261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:54.390133 20261 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fec72677884217a81035f48bd5afc4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33213 } }
I20260808 04:00:54.390689 20261 leader_election.cc:302] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2fec72677884217a81035f48bd5afc4; no voters:
I20260808 04:00:54.391289 20261 leader_election.cc:288] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:00:54.391453 20272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:00:54.392256 20272 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [term 1 LEADER]: Becoming Leader. State: Replica: d2fec72677884217a81035f48bd5afc4, State: Running, Role: LEADER
I20260808 04:00:54.392851 20272 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fec72677884217a81035f48bd5afc4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33213 } }
I20260808 04:00:54.393527 20261 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:00:54.395977 20273 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2fec72677884217a81035f48bd5afc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fec72677884217a81035f48bd5afc4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33213 } } }
I20260808 04:00:54.396391 20273 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:00:54.397117 20274 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2fec72677884217a81035f48bd5afc4. Latest consensus state: current_term: 1 leader_uuid: "d2fec72677884217a81035f48bd5afc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fec72677884217a81035f48bd5afc4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33213 } } }
I20260808 04:00:54.397532 20274 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:00:54.397871 20281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:00:54.403427 20281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:00:54.413941 20281 catalog_manager.cc:1347] Generated new cluster ID: c7ffb9f467ff4d4e9f5b29528d5d831c
I20260808 04:00:54.414108 20281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:00:54.427109 20281 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:00:54.428586 20281 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:00:54.441565 20281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2fec72677884217a81035f48bd5afc4: Generated new TSK 0
I20260808 04:00:54.442492 20281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:00:54.497975 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 20152
I20260808 04:00:54.499368 20253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:54.597487 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 20152
2026-08-08T04:00:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2346 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:00:54 dist-test-slave-pww0 krb5kdc[20298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:00:54 dist-test-slave-pww0 krb5kdc[20298](info): set up 2 sockets
Aug 08 04:00:54 dist-test-slave-pww0 krb5kdc[20298](info): commencing operation
krb5kdc: starting...
W20260808 04:00:55.876049 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.006s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:00:56 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:00:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:00:56.213795 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40685
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:40685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:00:56.610956 20320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:56.611304 20320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:56.611482 20320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:56.619050 20320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:00:56.619203 20320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:56.619279 20320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:56.619359 20320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:00:56.619422 20320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:00:56.629830 20320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:40685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:56.631361 20320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:56.632789 20320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:56.645380 20329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:56.645430 20335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:56.645349 20330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:56.646387 20320 server_base.cc:1034] running on GCE node
I20260808 04:00:56.647332 20320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:56.648972 20320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:56.650216 20320 hybrid_clock.cc:648] HybridClock initialized: now 1786161656650156 us; error 49 us; skew 500 ppm
I20260808 04:00:56.650633 20320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:56 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161656, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:56.654163 20320 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:00:56.655977 20320 webserver.cc:460] Webserver started at http://127.28.251.254:41773/ using document root <none> and password file <none>
I20260808 04:00:56.656749 20320 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:56.656880 20320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:56.657212 20320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:00:56.660626 20320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "6ea5a9e486f4426092570d4fafd621de"
format_stamp: "Formatted at 2026-08-08 04:00:56 on dist-test-slave-pww0"
server_key: "43a39348228f6943dc0bdf7f16fdebe1"
server_key_iv: "b8699e10d8c9ebeb81e4ad5fdebaf459"
server_key_version: "encryptionkey@0"
I20260808 04:00:56.661491 20320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "6ea5a9e486f4426092570d4fafd621de"
format_stamp: "Formatted at 2026-08-08 04:00:56 on dist-test-slave-pww0"
server_key: "43a39348228f6943dc0bdf7f16fdebe1"
server_key_iv: "b8699e10d8c9ebeb81e4ad5fdebaf459"
server_key_version: "encryptionkey@0"
I20260808 04:00:56.667119 20320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:00:56.671205 20352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:56.672955 20320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:00:56.673172 20320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "6ea5a9e486f4426092570d4fafd621de"
format_stamp: "Formatted at 2026-08-08 04:00:56 on dist-test-slave-pww0"
server_key: "43a39348228f6943dc0bdf7f16fdebe1"
server_key_iv: "b8699e10d8c9ebeb81e4ad5fdebaf459"
server_key_version: "encryptionkey@0"
I20260808 04:00:56.673410 20320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:56.692977 20320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:56.697769 20320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:56.698107 20320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:56.717659 20320 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:40685
I20260808 04:00:56.717681 20443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:40685 every 8 connection(s)
I20260808 04:00:56.719146 20320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:00:56.723317 20444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:00:56.729043 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 20320
I20260808 04:00:56.729259 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:00:56.729945 29679 external_mini_cluster.cc:1428] Setting key 6989b96208a54369f621f5553cd7c1cb
I20260808 04:00:56.735709 20444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Bootstrap starting.
I20260808 04:00:56.739704 20444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:56.740867 20444 log.cc:824] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:56.744225 20444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: No bootstrap required, opened a new log
Aug 08 04:00:56 dist-test-slave-pww0 krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:56.753656 20444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:00:56.754209 20444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:56.754329 20444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea5a9e486f4426092570d4fafd621de, State: Initialized, Role: FOLLOWER
I20260808 04:00:56.755033 20444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:00:56.755293 20444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:00:56.755440 20444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:00:56.755676 20444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:56.756012 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:56.732245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.28.251.254:40685)
0808 04:00:56.733160 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:56.733178 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:56.733235 (+ 57us) server_negotiation.cc:407] Connection header received
0808 04:00:56.735667 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:56.735707 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:56.736236 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:56.736724 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:56.738094 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:56.740084 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:56.741657 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:56.742487 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:56.742679 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:56.747400 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:56.747416 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:56.747444 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:56.747546 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:56.750958 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:56.752016 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:56.752028 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:56.752036 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:56.752250 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:56.752729 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:56.752733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:56.752735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:56.753377 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:56.753624 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:56.754223 (+ 599us) server_negotiation.cc:299] Negotiation successful
0808 04:00:56.754735 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":389,"threads_started":1}
I20260808 04:00:56.758231 20444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:00:56.758770 20444 leader_election.cc:302] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ea5a9e486f4426092570d4fafd621de; no voters:
I20260808 04:00:56.759414 20444 leader_election.cc:288] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:00:56.759557 20451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:00:56.760253 20451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 1 LEADER]: Becoming Leader. State: Replica: 6ea5a9e486f4426092570d4fafd621de, State: Running, Role: LEADER
I20260808 04:00:56.760878 20451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:00:56.761749 20444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:00:56.763881 20454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ea5a9e486f4426092570d4fafd621de. Latest consensus state: current_term: 1 leader_uuid: "6ea5a9e486f4426092570d4fafd621de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } } }
I20260808 04:00:56.763855 20452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ea5a9e486f4426092570d4fafd621de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } } }
I20260808 04:00:56.764335 20452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: This master's current role is: LEADER
I20260808 04:00:56.764335 20454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: This master's current role is: LEADER
I20260808 04:00:56.765257 20457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:00:56.770452 20457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:00:56.781373 20457 catalog_manager.cc:1347] Generated new cluster ID: af7bf47773d64c20aa9268092c42496f
I20260808 04:00:56.781507 20457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:00:56.802977 20457 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:00:56.805097 20457 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:00:56.822729 20457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Generated new TSK 0
I20260808 04:00:56.823742 20457 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 04:00:56.913201 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40685
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:00:57.312762 20485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:57.313078 20485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:57.313220 20485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:57.321144 20485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:57.321276 20485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:57.321455 20485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:00:57.332202 20485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:57.333653 20485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:57.335194 20485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:57.349334 20494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:57.349320 20499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:57.349246 20495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:57.350239 20485 server_base.cc:1034] running on GCE node
I20260808 04:00:57.350961 20485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:57.352079 20485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:57.353331 20485 hybrid_clock.cc:648] HybridClock initialized: now 1786161657353258 us; error 56 us; skew 500 ppm
I20260808 04:00:57.353710 20485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:57 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161657, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:57.357268 20485 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 04:00:57.358901 20485 webserver.cc:460] Webserver started at http://127.28.251.193:43303/ using document root <none> and password file <none>
I20260808 04:00:57.359699 20485 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:57.359833 20485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:57.360143 20485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:00:57.363435 20485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "5deea28f50b9480395a0ccddeeb78bfe"
format_stamp: "Formatted at 2026-08-08 04:00:57 on dist-test-slave-pww0"
server_key: "af1575d19b666299c124148575ded62a"
server_key_iv: "0ebafebf569dce5a482a9d431c174246"
server_key_version: "encryptionkey@0"
I20260808 04:00:57.364310 20485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "5deea28f50b9480395a0ccddeeb78bfe"
format_stamp: "Formatted at 2026-08-08 04:00:57 on dist-test-slave-pww0"
server_key: "af1575d19b666299c124148575ded62a"
server_key_iv: "0ebafebf569dce5a482a9d431c174246"
server_key_version: "encryptionkey@0"
I20260808 04:00:57.369925 20485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:00:57.373864 20510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:57.375396 20485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:00:57.375604 20485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "5deea28f50b9480395a0ccddeeb78bfe"
format_stamp: "Formatted at 2026-08-08 04:00:57 on dist-test-slave-pww0"
server_key: "af1575d19b666299c124148575ded62a"
server_key_iv: "0ebafebf569dce5a482a9d431c174246"
server_key_version: "encryptionkey@0"
I20260808 04:00:57.375821 20485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:57.390781 20485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:57.395201 20485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:57.395540 20485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:57.396610 20485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:00:57.398090 20485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:00:57.398173 20485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:57.398305 20485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:00:57.398375 20485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:57.431051 20485 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:35607
I20260808 04:00:57.431098 20702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:35607 every 8 connection(s)
I20260808 04:00:57.432430 20485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:00:57.433911 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 20485
I20260808 04:00:57.434130 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 04:00:57.434952 29679 external_mini_cluster.cc:1428] Setting key 853f5ffbb14c48b3eb0e3eaf5ff4fc00
Aug 08 04:00:57 dist-test-slave-pww0 krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161657, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:57.456053 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:57.435335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:45769 (local address 127.28.251.254:40685)
0808 04:00:57.435856 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:57.435861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:57.437095 (+ 1234us) server_negotiation.cc:407] Connection header received
0808 04:00:57.439133 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:57.439139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:57.439232 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:57.439332 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:57.441094 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:57.441870 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:57.443935 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:57.444521 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:57.444673 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:57.448884 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:57.448891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:57.448896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:57.448947 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:57.452228 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:57.453203 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:57.453210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:57.453213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:57.453387 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:57.453997 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:57.454000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:57.454002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:57.454301 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:57.454434 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:57.455505 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0808 04:00:57.455756 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":294,"threads_started":1}
I20260808 04:00:57.457523 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:57.436041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:45769)
0808 04:00:57.436912 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:57.436978 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:57.438812 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:57.439648 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:57.439660 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:57.440242 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:57.440903 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:57.440923 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:57.442035 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:57.442039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:57.443729 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:57.443745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:57.444772 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:57.444779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:57.445055 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:57.445827 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:57.445856 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:57.448599 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:57.452448 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:57.452457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:57.452474 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:57.452969 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:57.453542 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:57.453547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:57.453550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:57.453883 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:57.454435 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:57.454445 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:57.454961 (+ 516us) client_negotiation.cc:769] Sending connection context
0808 04:00:57.455289 (+ 328us) client_negotiation.cc:240] Negotiation successful
0808 04:00:57.456022 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":312,"threads_started":1}
I20260808 04:00:57.459429 20706 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40685
I20260808 04:00:57.459856 20706 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:57.460703 20706 heartbeater.cc:511] Master 127.28.251.254:40685 requested a full tablet report, sending...
I20260808 04:00:57.463205 20386 ts_manager.cc:194] Registered new tserver with Master: 5deea28f50b9480395a0ccddeeb78bfe (127.28.251.193:35607)
I20260808 04:00:57.464809 20386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:45769
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 04:00:57.509980 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40685
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:00:57.872824 20717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:57.873131 20717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:57.873258 20717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:57.881326 20717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:57.881458 20717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:57.881608 20717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:00:57.892004 20717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:57.893451 20717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:57.894884 20717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:57.908215 20726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:57.908169 20729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:57.908238 20736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:57.909057 20717 server_base.cc:1034] running on GCE node
I20260808 04:00:57.909682 20717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:57.910635 20717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:57.911844 20717 hybrid_clock.cc:648] HybridClock initialized: now 1786161657911781 us; error 50 us; skew 500 ppm
I20260808 04:00:57.912189 20717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:57 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161657, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:57.915422 20717 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 04:00:57.916910 20717 webserver.cc:460] Webserver started at http://127.28.251.194:44957/ using document root <none> and password file <none>
I20260808 04:00:57.917604 20717 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:57.917722 20717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:57.918016 20717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:00:57.921037 20717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "a7bc96cc672e4a61b1dc32183afc6648"
format_stamp: "Formatted at 2026-08-08 04:00:57 on dist-test-slave-pww0"
server_key: "10394f4ee5d2c39a32e071406a06672d"
server_key_iv: "e78470a625817b77501498d28b8a0ecb"
server_key_version: "encryptionkey@0"
I20260808 04:00:57.921833 20717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "a7bc96cc672e4a61b1dc32183afc6648"
format_stamp: "Formatted at 2026-08-08 04:00:57 on dist-test-slave-pww0"
server_key: "10394f4ee5d2c39a32e071406a06672d"
server_key_iv: "e78470a625817b77501498d28b8a0ecb"
server_key_version: "encryptionkey@0"
I20260808 04:00:57.927109 20717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:00:57.930850 20744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:57.932325 20717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:00:57.932483 20717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "a7bc96cc672e4a61b1dc32183afc6648"
format_stamp: "Formatted at 2026-08-08 04:00:57 on dist-test-slave-pww0"
server_key: "10394f4ee5d2c39a32e071406a06672d"
server_key_iv: "e78470a625817b77501498d28b8a0ecb"
server_key_version: "encryptionkey@0"
I20260808 04:00:57.932682 20717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:57.955564 20717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:57.960417 20717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:57.960734 20717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:57.961817 20717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:00:57.963444 20717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:00:57.963573 20717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 04:00:57.963709 20717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:00:57.963761 20717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:57.997406 20717 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:41725
I20260808 04:00:57.997453 20936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:41725 every 8 connection(s)
I20260808 04:00:57.998829 20717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:00:58.004580 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 20717
I20260808 04:00:58.004807 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 04:00:58.005407 29679 external_mini_cluster.cc:1428] Setting key 3a136564cff8e9b018ca5b6a402c4d07
Aug 08 04:00:58 dist-test-slave-pww0 krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161657, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:58.022109 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.001677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:54313 (local address 127.28.251.254:40685)
0808 04:00:58.002132 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.002136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.003138 (+ 1002us) server_negotiation.cc:407] Connection header received
0808 04:00:58.005596 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.005601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.005682 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.005809 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:58.007834 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.008566 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.010897 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.011355 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.011515 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.015904 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:58.015910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:58.015914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:58.015954 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:58.018650 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:58.019607 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:58.019611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:58.019614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:58.019754 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:58.020219 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:58.020223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:58.020225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:58.020524 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:58.020662 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.021481 (+ 819us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.021725 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":227,"threads_started":1}
I20260808 04:00:58.023489 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.002153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:54313)
0808 04:00:58.002940 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:58.002983 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:58.005211 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:58.006125 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:58.006135 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:58.006786 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:58.007623 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.007657 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.008738 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.008743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.010716 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.010733 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.011632 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.011639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.011931 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.012744 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:58.012777 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:58.015611 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:58.018837 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:58.018847 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:58.018864 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:58.019429 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:58.019878 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:58.019884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:58.019888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:58.020096 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:58.020661 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:58.020670 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:58.021067 (+ 397us) client_negotiation.cc:769] Sending connection context
0808 04:00:58.021330 (+ 263us) client_negotiation.cc:240] Negotiation successful
0808 04:00:58.021957 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":296,"threads_started":1}
I20260808 04:00:58.025139 20940 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40685
I20260808 04:00:58.025514 20940 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:58.026556 20940 heartbeater.cc:511] Master 127.28.251.254:40685 requested a full tablet report, sending...
I20260808 04:00:58.028497 20386 ts_manager.cc:194] Registered new tserver with Master: a7bc96cc672e4a61b1dc32183afc6648 (127.28.251.194:41725)
I20260808 04:00:58.029597 20386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:54313
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 04:00:58.081470 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40685
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:00:58.445406 20948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:58.445760 20948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:58.445873 20948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:58.454100 20948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:58.454253 20948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:58.454396 20948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:00:58.466934 20948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:58.468318 20706 heartbeater.cc:503] Master 127.28.251.254:40685 was elected leader, sending a full tablet report...
I20260808 04:00:58.468580 20948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:58.470261 20948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:58.484372 20962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:58.484511 20963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:58.484488 20965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:58.489490 20948 server_base.cc:1034] running on GCE node
I20260808 04:00:58.490268 20948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:58.491400 20948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:58.492677 20948 hybrid_clock.cc:648] HybridClock initialized: now 1786161658492620 us; error 44 us; skew 500 ppm
I20260808 04:00:58.493036 20948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:58 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161658, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:58.496702 20948 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 04:00:58.498576 20948 webserver.cc:460] Webserver started at http://127.28.251.195:33147/ using document root <none> and password file <none>
I20260808 04:00:58.499382 20948 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:58.499552 20948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:58.499884 20948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:00:58.503139 20948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "e96928aff4f647e1846ec372a5d7ed2c"
format_stamp: "Formatted at 2026-08-08 04:00:58 on dist-test-slave-pww0"
server_key: "0ccc6696947ba58e1b382cdc97dfe96d"
server_key_iv: "90442bdeba3c1042471193e4ee54a9ab"
server_key_version: "encryptionkey@0"
I20260808 04:00:58.504050 20948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "e96928aff4f647e1846ec372a5d7ed2c"
format_stamp: "Formatted at 2026-08-08 04:00:58 on dist-test-slave-pww0"
server_key: "0ccc6696947ba58e1b382cdc97dfe96d"
server_key_iv: "90442bdeba3c1042471193e4ee54a9ab"
server_key_version: "encryptionkey@0"
I20260808 04:00:58.510147 20948 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260808 04:00:58.514626 20979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:58.516253 20948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 04:00:58.516450 20948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "e96928aff4f647e1846ec372a5d7ed2c"
format_stamp: "Formatted at 2026-08-08 04:00:58 on dist-test-slave-pww0"
server_key: "0ccc6696947ba58e1b382cdc97dfe96d"
server_key_iv: "90442bdeba3c1042471193e4ee54a9ab"
server_key_version: "encryptionkey@0"
I20260808 04:00:58.516669 20948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:58.532791 20948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:00:58.537861 20948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:00:58.538164 20948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:00:58.539228 20948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:00:58.540819 20948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:00:58.540905 20948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:58.541049 20948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:00:58.541107 20948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:58.575207 20948 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:38979
I20260808 04:00:58.575260 21174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:38979 every 8 connection(s)
I20260808 04:00:58.576653 20948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:00:58.586349 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 20948
I20260808 04:00:58.586558 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 04:00:58.587011 29679 external_mini_cluster.cc:1428] Setting key 26e64cbcbe518fa4311206f6bdf5c347
Aug 08 04:00:58 dist-test-slave-pww0 krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161658, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:00:58.599449 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.579593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:52919 (local address 127.28.251.254:40685)
0808 04:00:58.580106 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.580111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.581167 (+ 1056us) server_negotiation.cc:407] Connection header received
0808 04:00:58.583101 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.583108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.583197 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.583311 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:58.585086 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.585887 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.587740 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.588267 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.588424 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.593265 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:58.593271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:58.593275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:58.593316 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:58.596112 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:58.597078 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:58.597083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:58.597086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:58.597238 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:58.597688 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:58.597691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:58.597693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:58.597991 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:58.598111 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.598911 (+ 800us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.599106 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":273,"threads_started":1}
I20260808 04:00:58.600569 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.580165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:52919)
0808 04:00:58.580995 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:58.581037 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:58.582777 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:58.583600 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:58.583609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:58.584239 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:00:58.584886 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.584907 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.586054 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.586058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.587546 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.587563 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.588552 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.588561 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.588837 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.590344 (+ 1507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:00:58.590385 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:00:58.592952 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:00:58.596325 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:58.596335 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:58.596360 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:58.596900 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:58.597358 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:00:58.597362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:00:58.597365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:00:58.597594 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:00:58.598102 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:00:58.598108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:00:58.598520 (+ 412us) client_negotiation.cc:769] Sending connection context
0808 04:00:58.598806 (+ 286us) client_negotiation.cc:240] Negotiation successful
0808 04:00:58.599275 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":316,"threads_started":1}
I20260808 04:00:58.602288 21176 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40685
I20260808 04:00:58.602615 21176 heartbeater.cc:464] Registering TS with master...
I20260808 04:00:58.603410 21176 heartbeater.cc:511] Master 127.28.251.254:40685 requested a full tablet report, sending...
I20260808 04:00:58.604934 20385 ts_manager.cc:194] Registered new tserver with Master: e96928aff4f647e1846ec372a5d7ed2c (127.28.251.195:38979)
I20260808 04:00:58.605834 20385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:52919
I20260808 04:00:58.616192 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:00:58.634079 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.621119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.28.251.254:40685)
0808 04:00:58.621364 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.621370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.621690 (+ 320us) server_negotiation.cc:407] Connection header received
0808 04:00:58.621894 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.621898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.621971 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.622056 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:58.623576 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.624354 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.625724 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.626159 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.626272 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.628894 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:58.628900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:58.628904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:58.628944 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:58.631488 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:58.632219 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:58.632223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:58.632225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:58.632370 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:58.632667 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:58.632670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:58.632672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:58.633005 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:58.633194 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.633586 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.633788 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 04:00:58.638676 20385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:00:58.646211 20385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:00:58.671556 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.659717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:38979 (local address 127.0.0.1:37372)
0808 04:00:58.660859 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:58.660915 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:58.661104 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:58.662615 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:58.662624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:58.662691 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:58.663198 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.663212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.665725 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.665731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.668325 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.668347 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.670473 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.670481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.670709 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.670771 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 04:00:58.670897 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 04:00:58.671110 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"threads_started":1}
I20260808 04:00:58.673403 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.660114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.28.251.195:38979)
0808 04:00:58.661889 (+ 1775us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.661899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.661939 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:00:58.662055 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.662063 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.662347 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.663509 (+ 1162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:58.663802 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.665540 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.668614 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.670192 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.672501 (+ 2309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.672725 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.672818 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.673005 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1577,"thread_start_us":1317,"threads_started":1}
I20260808 04:00:58.680954 21061 tablet_service.cc:1467] Processing CreateTablet for tablet 1185c1afa2bf40c192337f139fa29fbd (DEFAULT_TABLE table=test-table [id=eb7e069e418241deb4e97730b6d35aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:00:58.682591 21061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1185c1afa2bf40c192337f139fa29fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:00:58.682878 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.660106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41725 (local address 127.0.0.1:37380)
0808 04:00:58.672104 (+ 11998us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:58.672128 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:58.672248 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:58.675884 (+ 3636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:58.675889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:58.675922 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:58.676115 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.676126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.678126 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.678132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.680025 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.680045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.682055 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.682061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.682261 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.682299 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 04:00:58.682404 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 04:00:58.682555 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11860,"thread_start_us":341,"threads_started":1}
I20260808 04:00:58.683687 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.660716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.28.251.194:41725)
0808 04:00:58.675240 (+ 14524us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.675248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.675271 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:00:58.675354 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.675361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.675632 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.675870 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:58.676261 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.677972 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.680274 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.681841 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.682041 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.682255 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.683137 (+ 882us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.683326 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14344,"thread_start_us":14231,"threads_started":1}
I20260808 04:00:58.685161 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.662428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.28.251.193:35607)
0808 04:00:58.674385 (+ 11957us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.674396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.675164 (+ 768us) server_negotiation.cc:407] Connection header received
0808 04:00:58.675283 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.675303 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.675634 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.675870 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:58.677016 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.679620 (+ 2604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.682100 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.683501 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.683703 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.683880 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.684686 (+ 806us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.684835 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11747,"thread_start_us":11583,"threads_started":1}
I20260808 04:00:58.685798 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.661807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:35607 (local address 127.0.0.1:33444)
0808 04:00:58.674853 (+ 13046us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:58.674879 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:58.675022 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:58.676283 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:58.676287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:58.676311 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:58.676801 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.676815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.679785 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.679790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.681904 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.681920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.684037 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.684043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.684222 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.684256 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:00:58.684360 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 04:00:58.684499 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12925,"thread_start_us":12773,"threads_started":1}
I20260808 04:00:58.688030 20817 tablet_service.cc:1467] Processing CreateTablet for tablet 1185c1afa2bf40c192337f139fa29fbd (DEFAULT_TABLE table=test-table [id=eb7e069e418241deb4e97730b6d35aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:00:58.689306 20817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1185c1afa2bf40c192337f139fa29fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:00:58.689930 20598 tablet_service.cc:1467] Processing CreateTablet for tablet 1185c1afa2bf40c192337f139fa29fbd (DEFAULT_TABLE table=test-table [id=eb7e069e418241deb4e97730b6d35aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:00:58.691529 20598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1185c1afa2bf40c192337f139fa29fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:00:58.698488 21204 tablet_bootstrap.cc:492] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c: Bootstrap starting.
I20260808 04:00:58.702759 21204 tablet_bootstrap.cc:654] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:58.703523 21206 tablet_bootstrap.cc:492] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: Bootstrap starting.
I20260808 04:00:58.704349 21204 log.cc:824] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:58.708077 21206 tablet_bootstrap.cc:654] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:58.708551 21209 tablet_bootstrap.cc:492] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe: Bootstrap starting.
I20260808 04:00:58.709538 21206 log.cc:824] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:58.712145 21209 tablet_bootstrap.cc:654] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe: Neither blocks nor log segments found. Creating new log.
I20260808 04:00:58.712607 21204 tablet_bootstrap.cc:492] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c: No bootstrap required, opened a new log
I20260808 04:00:58.712949 21204 ts_tablet_manager.cc:1397] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260808 04:00:58.713277 21209 log.cc:824] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe: Log is configured to *not* fsync() on all Append() calls
I20260808 04:00:58.713456 21206 tablet_bootstrap.cc:492] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: No bootstrap required, opened a new log
I20260808 04:00:58.713757 21206 ts_tablet_manager.cc:1397] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260808 04:00:58.716141 21209 tablet_bootstrap.cc:492] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe: No bootstrap required, opened a new log
I20260808 04:00:58.716440 21209 ts_tablet_manager.cc:1397] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:00:58.724838 21204 raft_consensus.cc:348] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:00:58.724867 21206 raft_consensus.cc:348] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:00:58.725384 21206 raft_consensus.cc:374] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:58.725515 21206 raft_consensus.cc:729] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7bc96cc672e4a61b1dc32183afc6648, State: Initialized, Role: FOLLOWER
I20260808 04:00:58.725589 21204 raft_consensus.cc:374] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:58.725780 21204 raft_consensus.cc:729] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e96928aff4f647e1846ec372a5d7ed2c, State: Initialized, Role: FOLLOWER
I20260808 04:00:58.726419 21206 consensus_queue.cc:260] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:00:58.726514 21204 consensus_queue.cc:260] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:00:58.727596 21209 raft_consensus.cc:348] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:00:58.728128 21209 raft_consensus.cc:374] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:00:58.728251 21209 raft_consensus.cc:729] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5deea28f50b9480395a0ccddeeb78bfe, State: Initialized, Role: FOLLOWER
I20260808 04:00:58.729007 21209 consensus_queue.cc:260] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:00:58.732856 21209 ts_tablet_manager.cc:1428] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260808 04:00:58.741811 21176 heartbeater.cc:503] Master 127.28.251.254:40685 was elected leader, sending a full tablet report...
I20260808 04:00:58.741916 20940 heartbeater.cc:503] Master 127.28.251.254:40685 was elected leader, sending a full tablet report...
I20260808 04:00:58.742399 21204 ts_tablet_manager.cc:1428] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c: Time spent starting tablet: real 0.029s user 0.017s sys 0.011s
I20260808 04:00:58.742405 21206 ts_tablet_manager.cc:1428] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: Time spent starting tablet: real 0.028s user 0.008s sys 0.018s
I20260808 04:00:58.747650 21214 raft_consensus.cc:482] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:00:58.748041 21214 raft_consensus.cc:504] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:00:58.749720 21214 leader_election.cc:288] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7bc96cc672e4a61b1dc32183afc6648 (127.28.251.194:41725), 5deea28f50b9480395a0ccddeeb78bfe (127.28.251.193:35607)
W20260808 04:00:58.751111 20942 tablet.cc:2369] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:00:58.756373 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.750026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:36845 (local address 127.28.251.194:41725)
0808 04:00:58.750216 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.750221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.750242 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:00:58.750402 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.750407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.750487 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.750586 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:58.751090 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.752303 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.754283 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.755543 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.755660 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.755718 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.756051 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.756156 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:00:58.756676 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.749952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41725 (local address 127.28.251.195:36845)
0808 04:00:58.750130 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:58.750151 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:58.750265 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:58.750623 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:58.750628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:58.750664 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:58.750921 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.750935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.752433 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.752439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.754103 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.754116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.755688 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.755695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.755890 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.755928 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 04:00:58.756028 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:00:58.756174 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":37}
I20260808 04:00:58.756821 21215 raft_consensus.cc:482] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:00:58.757153 21215 raft_consensus.cc:504] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:00:58.757654 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.750307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:60787 (local address 127.28.251.193:35607)
0808 04:00:58.750526 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.750531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.751201 (+ 670us) server_negotiation.cc:407] Connection header received
0808 04:00:58.751339 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.751344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.751419 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.751531 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:58.752100 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.753119 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.755391 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.756647 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.756821 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.756922 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.757266 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.757417 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 04:00:58.757696 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.749952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:35607 (local address 127.28.251.195:60787)
0808 04:00:58.751107 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:58.751123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:58.751229 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:58.751612 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:58.751616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:58.751641 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:58.751916 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.751925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.753327 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.753331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.755182 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.755200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.756894 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.756901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.757068 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.757102 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:00:58.757202 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:00:58.757334 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":452,"thread_start_us":328,"threads_started":1}
I20260808 04:00:58.758225 20854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1185c1afa2bf40c192337f139fa29fbd" candidate_uuid: "e96928aff4f647e1846ec372a5d7ed2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7bc96cc672e4a61b1dc32183afc6648" is_pre_election: true
I20260808 04:00:58.758854 20854 raft_consensus.cc:2454] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e96928aff4f647e1846ec372a5d7ed2c in term 0.
I20260808 04:00:58.758746 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1185c1afa2bf40c192337f139fa29fbd" candidate_uuid: "e96928aff4f647e1846ec372a5d7ed2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5deea28f50b9480395a0ccddeeb78bfe" is_pre_election: true
I20260808 04:00:58.759639 20988 leader_election.cc:302] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7bc96cc672e4a61b1dc32183afc6648, e96928aff4f647e1846ec372a5d7ed2c; no voters:
I20260808 04:00:58.759913 20625 raft_consensus.cc:2454] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e96928aff4f647e1846ec372a5d7ed2c in term 0.
I20260808 04:00:58.760118 21214 raft_consensus.cc:2781] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:00:58.760272 21214 raft_consensus.cc:482] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:00:58.760366 21214 raft_consensus.cc:3037] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:58.763305 21214 raft_consensus.cc:504] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:00:58.763963 21214 leader_election.cc:288] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [CANDIDATE]: Term 1 election: Requested vote from peers a7bc96cc672e4a61b1dc32183afc6648 (127.28.251.194:41725), 5deea28f50b9480395a0ccddeeb78bfe (127.28.251.193:35607)
I20260808 04:00:58.764495 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1185c1afa2bf40c192337f139fa29fbd" candidate_uuid: "e96928aff4f647e1846ec372a5d7ed2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5deea28f50b9480395a0ccddeeb78bfe"
I20260808 04:00:58.764528 20854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1185c1afa2bf40c192337f139fa29fbd" candidate_uuid: "e96928aff4f647e1846ec372a5d7ed2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7bc96cc672e4a61b1dc32183afc6648"
I20260808 04:00:58.764698 20625 raft_consensus.cc:3037] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:58.764804 20854 raft_consensus.cc:3037] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:00:58.765664 21215 leader_election.cc:288] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e96928aff4f647e1846ec372a5d7ed2c (127.28.251.195:38979), a7bc96cc672e4a61b1dc32183afc6648 (127.28.251.194:41725)
I20260808 04:00:58.768183 20625 raft_consensus.cc:2454] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e96928aff4f647e1846ec372a5d7ed2c in term 1.
I20260808 04:00:58.768448 20854 raft_consensus.cc:2454] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e96928aff4f647e1846ec372a5d7ed2c in term 1.
I20260808 04:00:58.769019 20988 leader_election.cc:302] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7bc96cc672e4a61b1dc32183afc6648, e96928aff4f647e1846ec372a5d7ed2c; no voters:
I20260808 04:00:58.769413 21214 raft_consensus.cc:2781] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:00:58.770077 21214 raft_consensus.cc:686] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 1 LEADER]: Becoming Leader. State: Replica: e96928aff4f647e1846ec372a5d7ed2c, State: Running, Role: LEADER
I20260808 04:00:58.770815 21214 consensus_queue.cc:237] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:00:58.777271 20385 catalog_manager.cc:5495] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c reported cstate change: term changed from 0 to 1, leader changed from <none> to e96928aff4f647e1846ec372a5d7ed2c (127.28.251.195). New cstate: current_term: 1 leader_uuid: "e96928aff4f647e1846ec372a5d7ed2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } health_report { overall_health: UNKNOWN } } }
I20260808 04:00:58.779935 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.761758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:53123 (local address 127.28.251.195:38979)
0808 04:00:58.761955 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.761961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.772315 (+ 10354us) server_negotiation.cc:407] Connection header received
0808 04:00:58.772712 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.772718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.772810 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.772884 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:58.773618 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.774790 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.777555 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.778863 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.779046 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.779128 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.779507 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.779653 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:00:58.780064 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.761550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:38979 (local address 127.28.251.193:53123)
0808 04:00:58.772176 (+ 10626us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:58.772204 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:58.772508 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:58.773084 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:58.773088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:58.773112 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:58.773472 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.773482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.774960 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.774965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.777371 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.777388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.779104 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.779111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.779278 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.779316 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 04:00:58.779437 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 04:00:58.779618 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10449,"mutex_wait_us":45,"thread_start_us":11488,"threads_started":1}
I20260808 04:00:58.780323 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.766583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:41655 (local address 127.28.251.194:41725)
0808 04:00:58.766779 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.766785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.772944 (+ 6159us) server_negotiation.cc:407] Connection header received
0808 04:00:58.773088 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.773094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.773176 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.773273 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:00:58.774132 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.775288 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.777773 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.778992 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.779125 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.779192 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.779886 (+ 694us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.780040 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:00:58.781999 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.765998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41725 (local address 127.28.251.193:41655)
0808 04:00:58.772821 (+ 6823us) negotiation.cc:107] Waiting for socket to connect
0808 04:00:58.772838 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:00:58.772962 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:00:58.773472 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:00:58.773477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:00:58.773505 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:00:58.773930 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.773942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.775535 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.775539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.777571 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:00:58.777587 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.779421 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.779428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:00:58.779645 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.779679 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:00:58.781119 (+ 1440us) client_negotiation.cc:240] Negotiation successful
0808 04:00:58.781272 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6682,"mutex_wait_us":38,"thread_start_us":5788,"threads_started":1}
I20260808 04:00:58.781997 21091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1185c1afa2bf40c192337f139fa29fbd" candidate_uuid: "5deea28f50b9480395a0ccddeeb78bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e96928aff4f647e1846ec372a5d7ed2c" is_pre_election: true
I20260808 04:00:58.782837 20854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1185c1afa2bf40c192337f139fa29fbd" candidate_uuid: "5deea28f50b9480395a0ccddeeb78bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7bc96cc672e4a61b1dc32183afc6648" is_pre_election: true
I20260808 04:00:58.783182 20854 raft_consensus.cc:2379] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5deea28f50b9480395a0ccddeeb78bfe in current term 1: Already voted for candidate e96928aff4f647e1846ec372a5d7ed2c in this term.
I20260808 04:00:58.783974 20513 leader_election.cc:302] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5deea28f50b9480395a0ccddeeb78bfe; no voters: a7bc96cc672e4a61b1dc32183afc6648, e96928aff4f647e1846ec372a5d7ed2c
I20260808 04:00:58.784373 21215 raft_consensus.cc:2726] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 04:00:58.794862 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.789017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.28.251.195:38979)
0808 04:00:58.789202 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.789209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.789266 (+ 57us) server_negotiation.cc:407] Connection header received
0808 04:00:58.789518 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.789521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.789620 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.789760 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:00:58.790175 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.790967 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.792516 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.792939 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.793058 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.793655 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:00:58.794201 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:00:58.794360 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.794460 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.794553 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:00:58.803344 20854 raft_consensus.cc:1261] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648 [term 1 FOLLOWER]: Refusing update from remote peer e96928aff4f647e1846ec372a5d7ed2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:00:58.803388 20625 raft_consensus.cc:1261] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 1 FOLLOWER]: Refusing update from remote peer e96928aff4f647e1846ec372a5d7ed2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:00:58.804354 21220 consensus_queue.cc:1035] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:00:58.804764 21214 consensus_queue.cc:1035] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:00:58.817608 21226 mvcc.cc:205] Tried to move back new op lower bound from 7316118154448470016 to 7316118154332217344. Current Snapshot: MvccSnapshot[applied={T|T < 7316118154448470016}]
I20260808 04:00:58.838248 20383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:00:58.838604 20383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:00:58.842686 20383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Sending DeleteTablet for 3 replicas of tablet 1185c1afa2bf40c192337f139fa29fbd
I20260808 04:00:58.844091 20817 tablet_service.cc:1514] Processing DeleteTablet for tablet 1185c1afa2bf40c192337f139fa29fbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:00:58 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:37380
I20260808 04:00:58.844220 20598 tablet_service.cc:1514] Processing DeleteTablet for tablet 1185c1afa2bf40c192337f139fa29fbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:00:58 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:33444
I20260808 04:00:58.845032 21243 tablet_replica.cc:330] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: stopping tablet replica
I20260808 04:00:58.845010 21061 tablet_service.cc:1514] Processing DeleteTablet for tablet 1185c1afa2bf40c192337f139fa29fbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:00:58 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:37372
I20260808 04:00:58.845379 21243 raft_consensus.cc:2229] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:00:58.845927 21243 raft_consensus.cc:2258] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:00:58.846201 21246 tablet_replica.cc:330] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c: stopping tablet replica
I20260808 04:00:58.846395 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 20485
I20260808 04:00:58.847080 21245 tablet_replica.cc:330] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe: stopping tablet replica
I20260808 04:00:58.847529 21245 raft_consensus.cc:2229] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:00:58.848076 21245 raft_consensus.cc:2258] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:00:58.848697 21243 ts_tablet_manager.cc:1905] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:00:58.850368 21245 ts_tablet_manager.cc:1905] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:00:58.851701 21246 raft_consensus.cc:2229] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:00:58.852614 21246 raft_consensus.cc:2258] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:00:58.857662 21243 ts_tablet_manager.cc:1918] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:00:58.857662 21245 ts_tablet_manager.cc:1918] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:00:58.857883 21245 log.cc:1196] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/1185c1afa2bf40c192337f139fa29fbd
I20260808 04:00:58.858536 21245 ts_tablet_manager.cc:1939] T 1185c1afa2bf40c192337f139fa29fbd P 5deea28f50b9480395a0ccddeeb78bfe: Deleting consensus metadata
I20260808 04:00:58.860662 20359 catalog_manager.cc:4841] TS 5deea28f50b9480395a0ccddeeb78bfe (127.28.251.193:35607): tablet 1185c1afa2bf40c192337f139fa29fbd (table test-table [id=eb7e069e418241deb4e97730b6d35aa6]) successfully deleted
I20260808 04:00:58.860764 21243 log.cc:1196] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/1185c1afa2bf40c192337f139fa29fbd
I20260808 04:00:58.861140 21246 ts_tablet_manager.cc:1905] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:00:58.861354 21243 ts_tablet_manager.cc:1939] T 1185c1afa2bf40c192337f139fa29fbd P a7bc96cc672e4a61b1dc32183afc6648: Deleting consensus metadata
I20260808 04:00:58.863438 20354 catalog_manager.cc:4841] TS a7bc96cc672e4a61b1dc32183afc6648 (127.28.251.194:41725): tablet 1185c1afa2bf40c192337f139fa29fbd (table test-table [id=eb7e069e418241deb4e97730b6d35aa6]) successfully deleted
I20260808 04:00:58.868518 21246 ts_tablet_manager.cc:1918] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:00:58.868718 21246 log.cc:1196] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/1185c1afa2bf40c192337f139fa29fbd
I20260808 04:00:58.869225 21246 ts_tablet_manager.cc:1939] T 1185c1afa2bf40c192337f139fa29fbd P e96928aff4f647e1846ec372a5d7ed2c: Deleting consensus metadata
Aug 08 04:00:58 dist-test-slave-pww0 krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 04:00:58.870957 20353 catalog_manager.cc:4841] TS e96928aff4f647e1846ec372a5d7ed2c (127.28.251.195:38979): tablet 1185c1afa2bf40c192337f139fa29fbd (table test-table [id=eb7e069e418241deb4e97730b6d35aa6]) successfully deleted
I20260808 04:00:58.878043 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:58.847655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.28.251.193:35607)
0808 04:00:58.847851 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:58.847857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:58.857744 (+ 9887us) server_negotiation.cc:407] Connection header received
0808 04:00:58.858335 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:58.858342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:58.858436 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:58.858529 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:58.861542 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.862705 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.864864 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:58.865490 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:58.865638 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:58.871383 (+ 5745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:58.871400 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:58.871417 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:58.871532 (+ 115us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:58.874574 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:58.875612 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:58.875620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:58.875623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:58.875776 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:58.876218 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:58.876221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:58.876223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:58.876695 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:58.876946 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:58.877428 (+ 482us) server_negotiation.cc:299] Negotiation successful
0808 04:00:58.877702 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:00:58.878877 20697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:58.972767 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 20485
I20260808 04:00:59.014376 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 20717
Aug 08 04:00:59 dist-test-slave-pww0 krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 04:00:59.039933 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:59.015397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.28.251.194:41725)
0808 04:00:59.015633 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:59.015641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:59.015737 (+ 96us) server_negotiation.cc:407] Connection header received
0808 04:00:59.016071 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:59.016077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:59.016167 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:59.016286 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:59.018171 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:59.019396 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:59.021358 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:59.022176 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:59.022338 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:59.032081 (+ 9743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:59.032105 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:59.032126 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:59.032189 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:59.036353 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:59.037397 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:59.037409 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:59.037413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:59.037606 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:59.038033 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:59.038039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:59.038042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:59.038570 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:59.038937 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:59.039223 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 04:00:59.039548 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 04:00:59.040778 20931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:59.185647 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 20717
I20260808 04:00:59.210256 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 20948
Aug 08 04:00:59 dist-test-slave-pww0 krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 04:00:59.229806 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:00:59.211229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.28.251.195:38979)
0808 04:00:59.211444 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 04:00:59.211448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:00:59.211497 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:00:59.211868 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:00:59.211872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:00:59.211943 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:00:59.212040 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:00:59.213868 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:59.214654 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:59.216644 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:00:59.217127 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:00:59.217233 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:00:59.223134 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:00:59.223154 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:00:59.223175 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:00:59.223236 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:00:59.226526 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:59.227557 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:59.227567 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:59.227570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:59.227765 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:00:59.228226 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:00:59.228229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:00:59.228231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:00:59.228652 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:00:59.228914 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 04:00:59.229271 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 04:00:59.229487 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 04:00:59.230679 21167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:59.362612 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 20948
I20260808 04:00:59.394830 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 20320
I20260808 04:00:59.396047 20435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:00:59.510521 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 20320
I20260808 04:00:59.541663 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40685
--webserver_interface=127.28.251.254
--webserver_port=41773
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:40685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:00:59.899395 21258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:00:59.899791 21258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:00:59.899900 21258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:00:59.907500 21258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:00:59.907641 21258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:00:59.907703 21258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:00:59.907763 21258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:00:59.907806 21258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:00:59.918255 21258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:40685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=41773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:00:59.919739 21258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:00:59.921058 21258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:00:59.949901 21271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:59.949867 21268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:00:59.949924 21273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:00:59.950654 21258 server_base.cc:1034] running on GCE node
I20260808 04:00:59.951829 21258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:00:59.953679 21258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:00:59.954913 21258 hybrid_clock.cc:648] HybridClock initialized: now 1786161659954857 us; error 48 us; skew 500 ppm
I20260808 04:00:59.955340 21258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:00:59 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161659, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:00:59.962350 21258 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:00:59.964538 21258 webserver.cc:460] Webserver started at http://127.28.251.254:41773/ using document root <none> and password file <none>
I20260808 04:00:59.965409 21258 fs_manager.cc:359] Metadata directory not provided
I20260808 04:00:59.965583 21258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:00:59.972725 21258 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 04:00:59.976536 21282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:00:59.978283 21258 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:00:59.978497 21258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "6ea5a9e486f4426092570d4fafd621de"
format_stamp: "Formatted at 2026-08-08 04:00:56 on dist-test-slave-pww0"
server_key: "43a39348228f6943dc0bdf7f16fdebe1"
server_key_iv: "b8699e10d8c9ebeb81e4ad5fdebaf459"
server_key_version: "encryptionkey@0"
I20260808 04:00:59.979791 21258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:00:59.997777 21258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:00.002867 21258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:00.003152 21258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:00.023221 21258 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:40685
I20260808 04:01:00.023267 21375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:40685 every 8 connection(s)
I20260808 04:01:00.025010 21258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:01:00.033020 21378 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:01:00.034824 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 21258
I20260808 04:01:00.035708 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:35607
--local_ip_for_outbound_sockets=127.28.251.193
--tserver_master_addrs=127.28.251.254:40685
--webserver_port=43303
--webserver_interface=127.28.251.193
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:01:00.037144 21378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Bootstrap starting.
I20260808 04:01:00.059741 21378 log.cc:824] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:00.073725 21378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:01:00.074309 21378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Bootstrap complete.
I20260808 04:01:00.084081 21378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:01:00.085214 21378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea5a9e486f4426092570d4fafd621de, State: Initialized, Role: FOLLOWER
I20260808 04:01:00.085914 21378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:01:00.086166 21378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:01:00.086293 21378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:01:00.086463 21378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:01:00.090564 21378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:01:00.091161 21378 leader_election.cc:302] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ea5a9e486f4426092570d4fafd621de; no voters:
I20260808 04:01:00.091866 21378 leader_election.cc:288] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:01:00.092033 21384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:01:00.093245 21384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 2 LEADER]: Becoming Leader. State: Replica: 6ea5a9e486f4426092570d4fafd621de, State: Running, Role: LEADER
I20260808 04:01:00.093920 21384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:01:00.094331 21378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:01:00.096830 21386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ea5a9e486f4426092570d4fafd621de. Latest consensus state: current_term: 2 leader_uuid: "6ea5a9e486f4426092570d4fafd621de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } } }
I20260808 04:01:00.097313 21386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: This master's current role is: LEADER
I20260808 04:01:00.097944 21385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6ea5a9e486f4426092570d4fafd621de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } } }
I20260808 04:01:00.098259 21385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: This master's current role is: LEADER
I20260808 04:01:00.103174 21394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:01:00.112882 21394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:01:00.114823 21394 catalog_manager.cc:1259] Loaded cluster ID: af7bf47773d64c20aa9268092c42496f
I20260808 04:01:00.115064 21394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:01:00.119367 21394 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 04:01:00.139256 21394 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 04:01:00.150584 21394 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:01:00.152887 21394 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Loaded TSK: 0
I20260808 04:01:00.154907 21394 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:01:00.435738 21382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:00.436070 21382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:00.436197 21382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:00.443966 21382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:00.444111 21382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:00.444271 21382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:01:00.454634 21382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:35607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=43303
--tserver_master_addrs=127.28.251.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:00.456216 21382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:00.457721 21382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:00.473982 21416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:00.473958 21417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:00.474613 21419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:00.475109 21382 server_base.cc:1034] running on GCE node
I20260808 04:01:00.476042 21382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:00.477414 21382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:00.478673 21382 hybrid_clock.cc:648] HybridClock initialized: now 1786161660478599 us; error 67 us; skew 500 ppm
I20260808 04:01:00.479074 21382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:00 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161660, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:00.483516 21382 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 04:01:00.485649 21382 webserver.cc:460] Webserver started at http://127.28.251.193:43303/ using document root <none> and password file <none>
I20260808 04:01:00.486662 21382 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:00.486847 21382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:00.494607 21382 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 04:01:00.498430 21426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:00.500344 21382 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:01:00.500563 21382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "5deea28f50b9480395a0ccddeeb78bfe"
format_stamp: "Formatted at 2026-08-08 04:00:57 on dist-test-slave-pww0"
server_key: "af1575d19b666299c124148575ded62a"
server_key_iv: "0ebafebf569dce5a482a9d431c174246"
server_key_version: "encryptionkey@0"
I20260808 04:01:00.501991 21382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:00.542045 21382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:00.547158 21382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:00.547435 21382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:00.548584 21382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:00.550088 21382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:00.550213 21382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:00.550309 21382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:00.550383 21382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:00.584508 21382 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:35607
I20260808 04:01:00.584599 21624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:35607 every 8 connection(s)
I20260808 04:01:00.586109 21382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:01:00.591620 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 21382
I20260808 04:01:00.592419 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:41725
--local_ip_for_outbound_sockets=127.28.251.194
--tserver_master_addrs=127.28.251.254:40685
--webserver_port=44957
--webserver_interface=127.28.251.194
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:01:00 dist-test-slave-pww0 krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161660, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:01:00.615447 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:00.589676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:41753)
0808 04:01:00.590690 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:00.590742 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:00.592913 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:00.595371 (+ 2458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:00.595380 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:00.596106 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:01:00.596813 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:00.596838 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:00.598642 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:00.598648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:00.600539 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:00.600561 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:00.601739 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:00.601747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:00.602041 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:00.602818 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:01:00.602856 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:01:00.605648 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:01:00.610556 (+ 4908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:00.610563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:00.610583 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:00.611120 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:00.611810 (+ 690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:00.611814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:00.611817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:00.612082 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:00.613239 (+ 1157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:01:00.613247 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:01:00.613603 (+ 356us) client_negotiation.cc:769] Sending connection context
0808 04:01:00.613884 (+ 281us) client_negotiation.cc:240] Negotiation successful
0808 04:01:00.614359 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":373,"threads_started":1}
I20260808 04:01:00.616144 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:00.589367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:41753 (local address 127.28.251.254:40685)
0808 04:01:00.589997 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:00.590011 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:00.590933 (+ 922us) server_negotiation.cc:407] Connection header received
0808 04:01:00.593557 (+ 2624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:00.593571 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:00.593970 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:00.594646 (+ 676us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:00.597096 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:00.598412 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:00.600745 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:00.601460 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:00.601627 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:00.606074 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:00.606084 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:00.606106 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:00.606159 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:00.610342 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:00.611314 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:00.611325 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:00.611333 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:00.611644 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:00.612268 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:00.612275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:00.612278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:00.613051 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:00.613284 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:00.614207 (+ 923us) server_negotiation.cc:299] Negotiation successful
0808 04:01:00.614793 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":290,"threads_started":1}
I20260808 04:01:00.619251 21625 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40685
I20260808 04:01:00.619782 21625 heartbeater.cc:464] Registering TS with master...
I20260808 04:01:00.620862 21625 heartbeater.cc:511] Master 127.28.251.254:40685 requested a full tablet report, sending...
I20260808 04:01:00.623634 21306 ts_manager.cc:194] Registered new tserver with Master: 5deea28f50b9480395a0ccddeeb78bfe (127.28.251.193:35607)
I20260808 04:01:00.625231 21306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:41753
W20260808 04:01:00.977352 21631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:00.977706 21631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:00.977854 21631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:00.985641 21631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:00.985795 21631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:00.985951 21631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:01:00.996780 21631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:41725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=44957
--tserver_master_addrs=127.28.251.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:00.998308 21631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:00.999848 21631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:01.014396 21647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:01.014351 21645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:01.014396 21644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:01.017269 21631 server_base.cc:1034] running on GCE node
I20260808 04:01:01.018083 21631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:01.019290 21631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:01.020540 21631 hybrid_clock.cc:648] HybridClock initialized: now 1786161661020484 us; error 76 us; skew 500 ppm
I20260808 04:01:01.020916 21631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:01 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161661, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:01.024441 21631 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 04:01:01.026145 21631 webserver.cc:460] Webserver started at http://127.28.251.194:44957/ using document root <none> and password file <none>
I20260808 04:01:01.026925 21631 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:01.027060 21631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:01.033107 21631 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 04:01:01.036322 21662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:01.037989 21631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:01:01.038184 21631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "a7bc96cc672e4a61b1dc32183afc6648"
format_stamp: "Formatted at 2026-08-08 04:00:57 on dist-test-slave-pww0"
server_key: "10394f4ee5d2c39a32e071406a06672d"
server_key_iv: "e78470a625817b77501498d28b8a0ecb"
server_key_version: "encryptionkey@0"
I20260808 04:01:01.039280 21631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:01.057731 21631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:01.062243 21631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:01.062498 21631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:01.063581 21631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:01.065050 21631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:01.065150 21631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:01.065241 21631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:01.065325 21631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:01.099188 21631 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:41725
I20260808 04:01:01.099232 21850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:41725 every 8 connection(s)
I20260808 04:01:01.100867 21631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:01:01.103130 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 21631
I20260808 04:01:01.103994 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:38979
--local_ip_for_outbound_sockets=127.28.251.195
--tserver_master_addrs=127.28.251.254:40685
--webserver_port=33147
--webserver_interface=127.28.251.195
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:01:01 dist-test-slave-pww0 krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161661, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:01:01.128136 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.103743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:48663 (local address 127.28.251.254:40685)
0808 04:01:01.103992 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:01.104000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:01.107911 (+ 3911us) server_negotiation.cc:407] Connection header received
0808 04:01:01.109912 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:01.109919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:01.110037 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:01.110185 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:01.112012 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.112996 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.114748 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.115402 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.115587 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.119587 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:01.119593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:01.119599 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:01.119649 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:01.122610 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:01.123942 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:01.123947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:01.123950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:01.124153 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:01.124956 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:01.124964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:01.124968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:01.125455 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:01.125646 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:01.126920 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0808 04:01:01.127404 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:01:01.128882 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.104368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:48663)
0808 04:01:01.107687 (+ 3319us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:01.107744 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:01.109585 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:01.110501 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:01.110512 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:01.111081 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:01:01.111779 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:01.111803 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.113214 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.113219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:01.114563 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:01.114578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.115690 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.115696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:01.115995 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.116706 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:01:01.116740 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:01:01.119283 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:01:01.122834 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:01.122846 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:01.122868 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:01.123749 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:01.124323 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:01.124329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:01.124333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:01.124658 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:01.125671 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:01:01.125685 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:01:01.126281 (+ 596us) client_negotiation.cc:769] Sending connection context
0808 04:01:01.126616 (+ 335us) client_negotiation.cc:240] Negotiation successful
0808 04:01:01.127246 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2961,"thread_start_us":2750,"threads_started":1}
I20260808 04:01:01.130903 21851 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40685
I20260808 04:01:01.131258 21851 heartbeater.cc:464] Registering TS with master...
I20260808 04:01:01.132308 21851 heartbeater.cc:511] Master 127.28.251.254:40685 requested a full tablet report, sending...
I20260808 04:01:01.134632 21306 ts_manager.cc:194] Registered new tserver with Master: a7bc96cc672e4a61b1dc32183afc6648 (127.28.251.194:41725)
I20260808 04:01:01.136073 21306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:48663
W20260808 04:01:01.474054 21856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:01.474365 21856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:01.474463 21856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:01.482262 21856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:01.482402 21856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:01.482522 21856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:01:01.493070 21856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:38979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=33147
--tserver_master_addrs=127.28.251.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:01.494504 21856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:01.496009 21856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:01.510198 21865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:01.510277 21866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:01.510237 21868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:01.511714 21856 server_base.cc:1034] running on GCE node
I20260808 04:01:01.512436 21856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:01.513530 21856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:01.514746 21856 hybrid_clock.cc:648] HybridClock initialized: now 1786161661514691 us; error 56 us; skew 500 ppm
I20260808 04:01:01.515103 21856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:01 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161661, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:01.518520 21856 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 04:01:01.520342 21856 webserver.cc:460] Webserver started at http://127.28.251.195:33147/ using document root <none> and password file <none>
I20260808 04:01:01.521363 21856 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:01.521550 21856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:01.528124 21856 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:01:01.531239 21876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:01.532944 21856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:01:01.533129 21856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "e96928aff4f647e1846ec372a5d7ed2c"
format_stamp: "Formatted at 2026-08-08 04:00:58 on dist-test-slave-pww0"
server_key: "0ccc6696947ba58e1b382cdc97dfe96d"
server_key_iv: "90442bdeba3c1042471193e4ee54a9ab"
server_key_version: "encryptionkey@0"
I20260808 04:01:01.534170 21856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:01.554822 21856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:01.559134 21856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:01.559408 21856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:01.560567 21856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:01.562083 21856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:01.562186 21856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:01.562312 21856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:01.562392 21856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:01.595851 21856 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:38979
I20260808 04:01:01.595906 22050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:38979 every 8 connection(s)
I20260808 04:01:01.597378 21856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:01:01.603057 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 21856
Aug 08 04:01:01 dist-test-slave-pww0 krb5kdc[20298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161661, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:01:01.620702 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.604076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.28.251.254:40685)
0808 04:01:01.604624 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:01.604629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:01.604970 (+ 341us) server_negotiation.cc:407] Connection header received
0808 04:01:01.605137 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:01.605141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:01.605200 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:01.605290 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:01.607205 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.608481 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.610292 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.610957 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.611055 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.614104 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:01.614111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:01.614116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:01.614160 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:01.617340 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:01.618302 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:01.618308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:01.618312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:01.618496 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:01.618974 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:01.618979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:01.618982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:01.619381 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:01.619634 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:01.620027 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 04:01:01.620264 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"spinlock_wait_cycles":7040,"thread_start_us":304,"threads_started":1}
I20260808 04:01:01.621356 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.600260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:53991 (local address 127.28.251.254:40685)
0808 04:01:01.600456 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:01.600461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:01.601760 (+ 1299us) server_negotiation.cc:407] Connection header received
0808 04:01:01.603879 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:01.603885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:01.603978 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:01.604099 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:01.605948 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.607026 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.609272 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.609969 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.610112 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.614287 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:01.614292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:01.614296 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:01.614344 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:01.617663 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:01.618589 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:01.618595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:01.618598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:01.618773 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:01.619310 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:01.619313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:01.619316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:01.619721 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:01.619863 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:01.620813 (+ 950us) server_negotiation.cc:299] Negotiation successful
0808 04:01:01.621037 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:01:01.622725 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.600820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:53991)
0808 04:01:01.601583 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:01.601635 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:01.603545 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:01.604437 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:01.604452 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:01.605130 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:01:01.605766 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:01.605787 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.607205 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.607211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:01.609102 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:01.609120 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.610222 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.610229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:01.610575 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.611423 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:01:01.611479 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:01:01.614011 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:01:01.617821 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:01.617830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:01.617846 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:01.618393 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:01.618898 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:01.618902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:01.618904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:01.619186 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:01.619868 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:01:01.619875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:01:01.620370 (+ 495us) client_negotiation.cc:769] Sending connection context
0808 04:01:01.620695 (+ 325us) client_negotiation.cc:240] Negotiation successful
0808 04:01:01.621304 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":297,"threads_started":1}
I20260808 04:01:01.624087 22052 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40685
I20260808 04:01:01.624409 22052 heartbeater.cc:464] Registering TS with master...
I20260808 04:01:01.625164 22052 heartbeater.cc:511] Master 127.28.251.254:40685 requested a full tablet report, sending...
I20260808 04:01:01.626678 21305 ts_manager.cc:194] Registered new tserver with Master: e96928aff4f647e1846ec372a5d7ed2c (127.28.251.195:38979)
I20260808 04:01:01.627665 21305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:53991
I20260808 04:01:01.629403 21625 heartbeater.cc:503] Master 127.28.251.254:40685 was elected leader, sending a full tablet report...
I20260808 04:01:01.636229 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:01:01.643328 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.637172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.28.251.254:40685)
0808 04:01:01.637415 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:01.637422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:01.637863 (+ 441us) server_negotiation.cc:407] Connection header received
0808 04:01:01.638059 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:01.638062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:01.638249 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:01.638382 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:01:01.638811 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.639717 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.641170 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.641607 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.641705 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.642201 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:01:01.642768 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:01:01.642888 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:01.642970 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 04:01:01.643046 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260808 04:01:01.644366 21305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:01:01.651906 21305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:01:01.689973 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.666294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.28.251.195:38979)
0808 04:01:01.681390 (+ 15096us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:01.681398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:01.681434 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:01:01.681534 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:01.681542 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:01.681793 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:01.682997 (+ 1204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:01.683655 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.685276 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.687682 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.688630 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.688820 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.689032 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:01.689481 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 04:01:01.689636 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14887,"thread_start_us":15887,"threads_started":1}
I20260808 04:01:01.690019 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.665212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41725 (local address 127.0.0.1:37394)
0808 04:01:01.673536 (+ 8324us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:01.673583 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:01.673777 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:01.681752 (+ 7975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:01.681763 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:01.681813 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:01.682374 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:01.682397 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.684413 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.684418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:01.687060 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:01.687077 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.689067 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.689075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:01.689234 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.689271 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:01:01.689337 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 04:01:01.689531 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8030,"mutex_wait_us":2,"thread_start_us":7687,"threads_started":1}
I20260808 04:01:01.690966 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.665463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:38979 (local address 127.0.0.1:37414)
0808 04:01:01.674197 (+ 8734us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:01.674216 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:01.674344 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:01.682851 (+ 8507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:01.682855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:01.683049 (+ 194us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:01.683418 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:01.683430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.685441 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.685451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:01.687514 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:01.687526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.688844 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.688851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:01.689090 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.689159 (+ 69us) client_negotiation.cc:769] Sending connection context
0808 04:01:01.689292 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 04:01:01.690555 (+ 1263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8587,"mutex_wait_us":1069,"spinlock_wait_cycles":326400,"thread_start_us":8433,"threads_started":1}
I20260808 04:01:01.691366 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.665459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.28.251.194:41725)
0808 04:01:01.680875 (+ 15416us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:01.680886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:01.680929 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:01:01.681059 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:01.681069 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:01.681414 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:01.681706 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:01.682653 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.684286 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.687251 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.688772 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.690702 (+ 1930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.690840 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:01.690894 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 04:01:01.691066 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15166,"thread_start_us":16469,"threads_started":1}
I20260808 04:01:01.692103 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.666096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:35607 (local address 127.0.0.1:33462)
0808 04:01:01.682529 (+ 16433us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:01.682550 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:01.683630 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:01.684633 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:01.684636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:01.684661 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:01.685056 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:01.685066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.686837 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.686842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:01.688926 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:01.688942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.691287 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.691295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:01.691496 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.691531 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:01:01.691633 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 04:01:01.691762 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16241,"mutex_wait_us":33,"thread_start_us":16316,"threads_started":1}
I20260808 04:01:01.694650 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:01.666210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.28.251.193:35607)
0808 04:01:01.683720 (+ 17510us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:01.683729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:01.683759 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:01:01.683852 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:01.683914 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:01.684131 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:01.684411 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:01.685234 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.686663 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.689139 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:01.690834 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:01.691035 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:01.691198 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:01.694079 (+ 2881us) server_negotiation.cc:299] Negotiation successful
0808 04:01:01.694263 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17316,"thread_start_us":17505,"threads_started":1}
I20260808 04:01:01.694717 21741 tablet_service.cc:1467] Processing CreateTablet for tablet 8db8ed147ddf400188ede8122df1d608 (DEFAULT_TABLE table=test-table [id=4bfc438975254c0aa50a7a365c2deb00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:01:01.694693 21957 tablet_service.cc:1467] Processing CreateTablet for tablet 8db8ed147ddf400188ede8122df1d608 (DEFAULT_TABLE table=test-table [id=4bfc438975254c0aa50a7a365c2deb00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:01:01.696220 21741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db8ed147ddf400188ede8122df1d608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:01:01.696280 21957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db8ed147ddf400188ede8122df1d608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:01:01.698972 21509 tablet_service.cc:1467] Processing CreateTablet for tablet 8db8ed147ddf400188ede8122df1d608 (DEFAULT_TABLE table=test-table [id=4bfc438975254c0aa50a7a365c2deb00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:01:01.700500 21509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db8ed147ddf400188ede8122df1d608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:01:01.709825 22081 tablet_bootstrap.cc:492] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: Bootstrap starting.
I20260808 04:01:01.712103 22082 tablet_bootstrap.cc:492] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: Bootstrap starting.
I20260808 04:01:01.715128 22081 tablet_bootstrap.cc:654] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: Neither blocks nor log segments found. Creating new log.
I20260808 04:01:01.715934 22082 tablet_bootstrap.cc:654] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: Neither blocks nor log segments found. Creating new log.
I20260808 04:01:01.716320 22083 tablet_bootstrap.cc:492] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: Bootstrap starting.
I20260808 04:01:01.717085 22081 log.cc:824] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:01.717223 22082 log.cc:824] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:01.719949 22083 tablet_bootstrap.cc:654] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: Neither blocks nor log segments found. Creating new log.
I20260808 04:01:01.720750 22082 tablet_bootstrap.cc:492] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: No bootstrap required, opened a new log
I20260808 04:01:01.721037 22083 log.cc:824] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:01.721071 22082 ts_tablet_manager.cc:1397] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260808 04:01:01.721585 22081 tablet_bootstrap.cc:492] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: No bootstrap required, opened a new log
I20260808 04:01:01.722234 22081 ts_tablet_manager.cc:1397] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260808 04:01:01.724715 22083 tablet_bootstrap.cc:492] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: No bootstrap required, opened a new log
I20260808 04:01:01.724990 22083 ts_tablet_manager.cc:1397] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 04:01:01.732187 22083 raft_consensus.cc:348] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:01:01.732234 22082 raft_consensus.cc:348] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:01:01.732632 22083 raft_consensus.cc:374] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:01:01.732714 22082 raft_consensus.cc:374] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:01:01.732738 22083 raft_consensus.cc:729] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5deea28f50b9480395a0ccddeeb78bfe, State: Initialized, Role: FOLLOWER
I20260808 04:01:01.732831 22082 raft_consensus.cc:729] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e96928aff4f647e1846ec372a5d7ed2c, State: Initialized, Role: FOLLOWER
I20260808 04:01:01.733454 22083 consensus_queue.cc:260] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:01:01.733464 22082 consensus_queue.cc:260] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:01:01.733626 22081 raft_consensus.cc:348] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:01:01.734098 22081 raft_consensus.cc:374] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:01:01.734221 22081 raft_consensus.cc:729] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7bc96cc672e4a61b1dc32183afc6648, State: Initialized, Role: FOLLOWER
I20260808 04:01:01.734951 22081 consensus_queue.cc:260] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:01:01.736186 22052 heartbeater.cc:503] Master 127.28.251.254:40685 was elected leader, sending a full tablet report...
I20260808 04:01:01.736411 22082 ts_tablet_manager.cc:1428] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: Time spent starting tablet: real 0.015s user 0.016s sys 0.001s
I20260808 04:01:01.736748 21851 heartbeater.cc:503] Master 127.28.251.254:40685 was elected leader, sending a full tablet report...
I20260808 04:01:01.738242 22083 ts_tablet_manager.cc:1428] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260808 04:01:01.738811 22081 ts_tablet_manager.cc:1428] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: Time spent starting tablet: real 0.016s user 0.008s sys 0.006s
W20260808 04:01:01.838724 21626 tablet.cc:2369] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:01:01.849463 22054 tablet.cc:2369] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:01:01.853068 21852 tablet.cc:2369] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:01:02.060640 22090 raft_consensus.cc:482] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:01:02.061028 22090 raft_consensus.cc:504] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:01:02.063097 22090 leader_election.cc:288] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7bc96cc672e4a61b1dc32183afc6648 (127.28.251.194:41725), e96928aff4f647e1846ec372a5d7ed2c (127.28.251.195:38979)
I20260808 04:01:02.070725 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:02.063842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:52507 (local address 127.28.251.195:38979)
0808 04:01:02.064205 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:02.064211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:02.064645 (+ 434us) server_negotiation.cc:407] Connection header received
0808 04:01:02.064879 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:02.064883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:02.064949 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:02.065030 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:02.065706 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.066482 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.068861 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.069704 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.069885 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:02.069974 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:02.070297 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 04:01:02.070458 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260808 04:01:02.070829 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:02.063659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:38979 (local address 127.28.251.193:52507)
0808 04:01:02.064501 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:02.064522 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:02.064723 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:02.065063 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:02.065067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:02.065094 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:02.065527 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:02.065539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.066646 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.066650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:02.068658 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:02.068675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.069947 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.069955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:02.070119 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:02.070150 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:01:02.070249 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 04:01:02.070430 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":423,"threads_started":1}
I20260808 04:01:02.071038 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:02.063688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:59199 (local address 127.28.251.194:41725)
0808 04:01:02.063998 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:02.064006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:02.064915 (+ 909us) server_negotiation.cc:407] Connection header received
0808 04:01:02.065088 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:02.065093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:02.065156 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:02.065236 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:02.065993 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.066775 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.068849 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.069994 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.070200 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:02.070281 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:02.070611 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 04:01:02.070769 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260808 04:01:02.071367 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:02.063662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41725 (local address 127.28.251.193:59199)
0808 04:01:02.064767 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:02.064785 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:02.064944 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:02.065294 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:02.065299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:02.065324 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:02.065725 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:02.065735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.066941 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.066946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:02.068627 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:02.068648 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.070283 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.070291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:02.070450 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:02.070475 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:01:02.070560 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:01:02.070702 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":138,"thread_start_us":592,"threads_started":1}
I20260808 04:01:02.072252 21991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8ed147ddf400188ede8122df1d608" candidate_uuid: "5deea28f50b9480395a0ccddeeb78bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e96928aff4f647e1846ec372a5d7ed2c" is_pre_election: true
I20260808 04:01:02.072517 21772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8ed147ddf400188ede8122df1d608" candidate_uuid: "5deea28f50b9480395a0ccddeeb78bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7bc96cc672e4a61b1dc32183afc6648" is_pre_election: true
I20260808 04:01:02.073045 21991 raft_consensus.cc:2454] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5deea28f50b9480395a0ccddeeb78bfe in term 0.
I20260808 04:01:02.073032 21772 raft_consensus.cc:2454] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5deea28f50b9480395a0ccddeeb78bfe in term 0.
I20260808 04:01:02.073742 21427 leader_election.cc:302] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5deea28f50b9480395a0ccddeeb78bfe, e96928aff4f647e1846ec372a5d7ed2c; no voters:
I20260808 04:01:02.074199 22090 raft_consensus.cc:2781] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:01:02.074373 22090 raft_consensus.cc:482] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:01:02.074488 22090 raft_consensus.cc:3037] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:01:02.076951 22090 raft_consensus.cc:504] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:01:02.077658 22090 leader_election.cc:288] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [CANDIDATE]: Term 1 election: Requested vote from peers a7bc96cc672e4a61b1dc32183afc6648 (127.28.251.194:41725), e96928aff4f647e1846ec372a5d7ed2c (127.28.251.195:38979)
I20260808 04:01:02.078337 21772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8ed147ddf400188ede8122df1d608" candidate_uuid: "5deea28f50b9480395a0ccddeeb78bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7bc96cc672e4a61b1dc32183afc6648"
I20260808 04:01:02.078336 21991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8ed147ddf400188ede8122df1d608" candidate_uuid: "5deea28f50b9480395a0ccddeeb78bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e96928aff4f647e1846ec372a5d7ed2c"
I20260808 04:01:02.078591 21772 raft_consensus.cc:3037] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:01:02.078593 21991 raft_consensus.cc:3037] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:01:02.082065 21991 raft_consensus.cc:2454] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5deea28f50b9480395a0ccddeeb78bfe in term 1.
I20260808 04:01:02.082065 21772 raft_consensus.cc:2454] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5deea28f50b9480395a0ccddeeb78bfe in term 1.
I20260808 04:01:02.082721 21428 leader_election.cc:302] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5deea28f50b9480395a0ccddeeb78bfe, a7bc96cc672e4a61b1dc32183afc6648; no voters:
I20260808 04:01:02.083134 22090 raft_consensus.cc:2781] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:01:02.083706 22090 raft_consensus.cc:686] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 1 LEADER]: Becoming Leader. State: Replica: 5deea28f50b9480395a0ccddeeb78bfe, State: Running, Role: LEADER
I20260808 04:01:02.084268 22090 consensus_queue.cc:237] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } } peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } }
I20260808 04:01:02.089370 21303 catalog_manager.cc:5495] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 5deea28f50b9480395a0ccddeeb78bfe (127.28.251.193). New cstate: current_term: 1 leader_uuid: "5deea28f50b9480395a0ccddeeb78bfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5deea28f50b9480395a0ccddeeb78bfe" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 35607 } health_report { overall_health: HEALTHY } } }
I20260808 04:01:02.154772 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:02.148462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.28.251.193:35607)
0808 04:01:02.148747 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:02.148754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:02.149068 (+ 314us) server_negotiation.cc:407] Connection header received
0808 04:01:02.149331 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:02.149336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:02.149435 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:02.149581 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:01:02.149998 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.150793 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.152423 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.152861 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.152978 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:02.153553 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:01:02.154124 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:01:02.154276 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:02.154390 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 04:01:02.154485 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 04:01:02.164310 21991 raft_consensus.cc:1261] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c [term 1 FOLLOWER]: Refusing update from remote peer 5deea28f50b9480395a0ccddeeb78bfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:01:02.164310 21772 raft_consensus.cc:1261] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648 [term 1 FOLLOWER]: Refusing update from remote peer 5deea28f50b9480395a0ccddeeb78bfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:01:02.165745 22100 consensus_queue.cc:1035] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e96928aff4f647e1846ec372a5d7ed2c" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 38979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:01:02.166138 22090 consensus_queue.cc:1035] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7bc96cc672e4a61b1dc32183afc6648" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:01:02.179903 22109 mvcc.cc:205] Tried to move back new op lower bound from 7316118168212189184 to 7316118167902007296. Current Snapshot: MvccSnapshot[applied={T|T < 7316118168212189184}]
I20260808 04:01:02.193060 21303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:01:02.193409 21303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:01:02.197430 21303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Sending DeleteTablet for 3 replicas of tablet 8db8ed147ddf400188ede8122df1d608
I20260808 04:01:02.198750 21741 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db8ed147ddf400188ede8122df1d608 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:01:02 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:37394
I20260808 04:01:02.198750 21509 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db8ed147ddf400188ede8122df1d608 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:01:02 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:33462
I20260808 04:01:02.199597 21957 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db8ed147ddf400188ede8122df1d608 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:01:02 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:37414
I20260808 04:01:02.200788 22127 tablet_replica.cc:330] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: stopping tablet replica
I20260808 04:01:02.200914 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 21382
I20260808 04:01:02.201411 22127 raft_consensus.cc:2229] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:01:02.202586 22127 raft_consensus.cc:2258] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:01:02.205132 22127 ts_tablet_manager.cc:1905] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:01:02.213258 22127 ts_tablet_manager.cc:1918] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:01:02.213593 22127 log.cc:1196] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/8db8ed147ddf400188ede8122df1d608
I20260808 04:01:02.214473 22127 ts_tablet_manager.cc:1939] T 8db8ed147ddf400188ede8122df1d608 P e96928aff4f647e1846ec372a5d7ed2c: Deleting consensus metadata
I20260808 04:01:02.216773 21283 catalog_manager.cc:4841] TS e96928aff4f647e1846ec372a5d7ed2c (127.28.251.195:38979): tablet 8db8ed147ddf400188ede8122df1d608 (table test-table [id=4bfc438975254c0aa50a7a365c2deb00]) successfully deleted
I20260808 04:01:02.223538 22124 tablet_replica.cc:330] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: stopping tablet replica
I20260808 04:01:02.224125 22124 raft_consensus.cc:2229] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:01:02.224869 22124 raft_consensus.cc:2258] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:01:02.225771 22126 tablet_replica.cc:330] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: stopping tablet replica
I20260808 04:01:02.226292 22126 raft_consensus.cc:2229] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:01:02.226861 22126 raft_consensus.cc:2258] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:01:02.227097 22124 ts_tablet_manager.cc:1905] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:01:02.229430 22126 ts_tablet_manager.cc:1905] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:01:02.234501 22124 ts_tablet_manager.cc:1918] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:01:02.234771 22124 log.cc:1196] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/8db8ed147ddf400188ede8122df1d608
I20260808 04:01:02.235541 22124 ts_tablet_manager.cc:1939] T 8db8ed147ddf400188ede8122df1d608 P 5deea28f50b9480395a0ccddeeb78bfe: Deleting consensus metadata
I20260808 04:01:02.237730 21287 catalog_manager.cc:4841] TS 5deea28f50b9480395a0ccddeeb78bfe (127.28.251.193:35607): tablet 8db8ed147ddf400188ede8122df1d608 (table test-table [id=4bfc438975254c0aa50a7a365c2deb00]) successfully deleted
I20260808 04:01:02.238112 22126 ts_tablet_manager.cc:1918] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:01:02.238343 22126 log.cc:1196] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/8db8ed147ddf400188ede8122df1d608
I20260808 04:01:02.239141 22126 ts_tablet_manager.cc:1939] T 8db8ed147ddf400188ede8122df1d608 P a7bc96cc672e4a61b1dc32183afc6648: Deleting consensus metadata
I20260808 04:01:02.241367 21284 catalog_manager.cc:4841] TS a7bc96cc672e4a61b1dc32183afc6648 (127.28.251.194:41725): tablet 8db8ed147ddf400188ede8122df1d608 (table test-table [id=4bfc438975254c0aa50a7a365c2deb00]) successfully deleted
I20260808 04:01:02.245818 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:02.201854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.28.251.193:35607)
0808 04:01:02.202065 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:02.202071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:02.220155 (+ 18084us) server_negotiation.cc:407] Connection header received
0808 04:01:02.220501 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:02.220507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:02.220648 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:02.220732 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:02.227367 (+ 6635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.228581 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.233690 (+ 5109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.234443 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.234571 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:02.238929 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:02.238947 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:02.238968 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:02.239022 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:02.242368 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:02.243234 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:02.243250 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:02.243253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:02.243487 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:02.243936 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:02.243942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:02.243945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:02.244433 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:02.244754 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:02.245130 (+ 376us) server_negotiation.cc:299] Negotiation successful
0808 04:01:02.245391 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:01:02.246619 21619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:02.350512 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 21382
I20260808 04:01:02.382086 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 21631
I20260808 04:01:02.397091 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:02.382911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.28.251.194:41725)
0808 04:01:02.383095 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:02.383100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:02.383121 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:01:02.383231 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:02.383235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:02.383301 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:02.383397 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:02.384918 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.385753 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.387238 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.387737 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.387850 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:02.390654 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:02.390674 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:02.390695 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:02.390737 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:02.393794 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:02.394667 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:02.394700 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:02.394702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:02.394891 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:02.395373 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:02.395378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:02.395398 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:02.395941 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:02.396188 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:02.396563 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 04:01:02.396779 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:01:02.397827 21840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:02.490873 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 21631
I20260808 04:01:02.516731 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 21856
I20260808 04:01:02.533231 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:02.517599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.28.251.195:38979)
0808 04:01:02.517789 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:02.517794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:02.517893 (+ 99us) server_negotiation.cc:407] Connection header received
0808 04:01:02.518099 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:02.518104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:02.518174 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:02.518259 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:02.519887 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.520641 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.522047 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:02.522457 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:02.522571 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:02.526143 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:02.526167 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:02.526192 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:02.526271 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:02.529785 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:02.530739 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:02.530748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:02.530751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:02.530939 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:02.531386 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:02.531391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:02.531393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:02.531886 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:02.532190 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:02.532600 (+ 410us) server_negotiation.cc:299] Negotiation successful
0808 04:01:02.532861 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:01:02.533989 22045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:02.625257 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 21856
I20260808 04:01:02.657174 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 21258
I20260808 04:01:02.658399 21367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:02.790235 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 21258
I20260808 04:01:02.826469 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40685
--webserver_interface=127.28.251.254
--webserver_port=41773
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:40685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:01:03.198318 22142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:03.198642 22142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:03.198762 22142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:03.206214 22142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:01:03.206350 22142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:03.206396 22142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:03.206450 22142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:01:03.206494 22142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:01:03.216722 22142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:40685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=41773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:03.218212 22142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:03.219713 22142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:03.231887 22151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:03.231910 22153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:03.231840 22157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:03.233094 22142 server_base.cc:1034] running on GCE node
I20260808 04:01:03.233952 22142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:03.235508 22142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:03.236759 22142 hybrid_clock.cc:648] HybridClock initialized: now 1786161663236701 us; error 41 us; skew 500 ppm
I20260808 04:01:03.237175 22142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:03 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161663, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:03.240384 22142 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:01:03.241933 22142 webserver.cc:460] Webserver started at http://127.28.251.254:41773/ using document root <none> and password file <none>
I20260808 04:01:03.242656 22142 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:03.242782 22142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:03.248713 22142 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:01:03.251732 22171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:03.253137 22142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:01:03.253321 22142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "6ea5a9e486f4426092570d4fafd621de"
format_stamp: "Formatted at 2026-08-08 04:00:56 on dist-test-slave-pww0"
server_key: "43a39348228f6943dc0bdf7f16fdebe1"
server_key_iv: "b8699e10d8c9ebeb81e4ad5fdebaf459"
server_key_version: "encryptionkey@0"
I20260808 04:01:03.254350 22142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:03.280120 22142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:03.284582 22142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:03.284852 22142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:03.303824 22142 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:40685
I20260808 04:01:03.303798 22246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:40685 every 8 connection(s)
I20260808 04:01:03.305513 22142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:01:03.310380 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 22142
I20260808 04:01:03.311228 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:35607
--local_ip_for_outbound_sockets=127.28.251.193
--tserver_master_addrs=127.28.251.254:40685
--webserver_port=43303
--webserver_interface=127.28.251.193
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:01:03.313210 22247 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:01:03.316829 22247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Bootstrap starting.
I20260808 04:01:03.342797 22247 log.cc:824] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:03.366457 22247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:01:03.367254 22247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de: Bootstrap complete.
I20260808 04:01:03.381081 22247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:01:03.382442 22247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea5a9e486f4426092570d4fafd621de, State: Initialized, Role: FOLLOWER
I20260808 04:01:03.383268 22247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:01:03.383598 22247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:01:03.383764 22247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:01:03.383952 22247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:01:03.389899 22247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:01:03.390604 22247 leader_election.cc:302] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ea5a9e486f4426092570d4fafd621de; no voters:
I20260808 04:01:03.391387 22247 leader_election.cc:288] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 04:01:03.391623 22255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:01:03.392728 22255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [term 3 LEADER]: Becoming Leader. State: Replica: 6ea5a9e486f4426092570d4fafd621de, State: Running, Role: LEADER
I20260808 04:01:03.393322 22255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } }
I20260808 04:01:03.393910 22247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:01:03.395882 22257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ea5a9e486f4426092570d4fafd621de. Latest consensus state: current_term: 3 leader_uuid: "6ea5a9e486f4426092570d4fafd621de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } } }
I20260808 04:01:03.396248 22257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: This master's current role is: LEADER
I20260808 04:01:03.396695 22256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6ea5a9e486f4426092570d4fafd621de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ea5a9e486f4426092570d4fafd621de" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40685 } } }
I20260808 04:01:03.396996 22256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ea5a9e486f4426092570d4fafd621de [sys.catalog]: This master's current role is: LEADER
I20260808 04:01:03.408329 22267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:01:03.418286 22267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:01:03.420112 22267 catalog_manager.cc:1259] Loaded cluster ID: af7bf47773d64c20aa9268092c42496f
I20260808 04:01:03.420313 22267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 04:01:03.422582 22267 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f57739549ad google::LogMessage::Fail() at ??:0
@ 0x7f577395788a google::LogMessage::SendToLog() at ??:0
@ 0x7f57739544de google::LogMessage::Flush() at ??:0
@ 0x7f5773955a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260808 04:01:03.432053 22284 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f5789387512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5789384976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f578940c610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f578940c439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f57765a0345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f57765a391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f57765a3799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f577657eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f57798586db start_thread at ??:0
@ 0x7f57727ee61f clone at ??:0
W20260808 04:01:03.763401 22250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:03.763775 22250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:03.763906 22250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:03.771478 22250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:03.771620 22250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:03.771795 22250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:01:03.782042 22250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:35607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=43303
--tserver_master_addrs=127.28.251.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:03.783564 22250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:03.784960 22250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:03.798604 22315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:03.798630 22316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:03.798557 22321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:03.799582 22250 server_base.cc:1034] running on GCE node
I20260808 04:01:03.800237 22250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:03.801275 22250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:03.802520 22250 hybrid_clock.cc:648] HybridClock initialized: now 1786161663802468 us; error 64 us; skew 500 ppm
I20260808 04:01:03.802881 22250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:03 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161663, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:03.806241 22250 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 04:01:03.807874 22250 webserver.cc:460] Webserver started at http://127.28.251.193:43303/ using document root <none> and password file <none>
I20260808 04:01:03.808626 22250 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:03.808763 22250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:03.814963 22250 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260808 04:01:03.817991 22332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:03.819744 22250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:01:03.819931 22250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "5deea28f50b9480395a0ccddeeb78bfe"
format_stamp: "Formatted at 2026-08-08 04:00:57 on dist-test-slave-pww0"
server_key: "af1575d19b666299c124148575ded62a"
server_key_iv: "0ebafebf569dce5a482a9d431c174246"
server_key_version: "encryptionkey@0"
I20260808 04:01:03.820981 22250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:03.840978 22250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:03.845808 22250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:03.846113 22250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:03.847240 22250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:03.848752 22250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:03.848882 22250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:03.849000 22250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:03.849072 22250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:03.882659 22250 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:35607
I20260808 04:01:03.882735 22508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:35607 every 8 connection(s)
I20260808 04:01:03.884478 22250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:01:03.889580 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:03.887580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:60541)
0808 04:01:03.888378 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:03.888612 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":302,"threads_started":1}
W20260808 04:01:03.890322 22509 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:40685 (0 consecutive failures): Network error: Failed to ping master at 127.28.251.254:40685: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
I20260808 04:01:03.891172 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:03.890837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:59979)
0808 04:01:03.890989 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:03.891062 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 04:01:03.892059 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:03.891718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:38041)
0808 04:01:03.891856 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:03.891941 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:01:03.892299 22509 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:01:03.893800 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 22250
I20260808 04:01:03.894450 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:41725
--local_ip_for_outbound_sockets=127.28.251.194
--tserver_master_addrs=127.28.251.254:40685
--webserver_port=44957
--webserver_interface=127.28.251.194
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:01:04.261627 22514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:04.261953 22514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:04.262058 22514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:04.269735 22514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:04.269882 22514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:04.270006 22514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:01:04.281080 22514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:41725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=44957
--tserver_master_addrs=127.28.251.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:04.282502 22514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:04.284024 22514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:04.298142 22526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:04.298192 22531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:04.298395 22527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:04.299654 22514 server_base.cc:1034] running on GCE node
I20260808 04:01:04.300395 22514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:04.301472 22514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:04.302728 22514 hybrid_clock.cc:648] HybridClock initialized: now 1786161664302677 us; error 46 us; skew 500 ppm
I20260808 04:01:04.303076 22514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:04 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161664, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:04.306505 22514 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 04:01:04.308248 22514 webserver.cc:460] Webserver started at http://127.28.251.194:44957/ using document root <none> and password file <none>
I20260808 04:01:04.309013 22514 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:04.309149 22514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:04.315220 22514 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:01:04.318543 22540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:04.320106 22514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:01:04.320281 22514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "a7bc96cc672e4a61b1dc32183afc6648"
format_stamp: "Formatted at 2026-08-08 04:00:57 on dist-test-slave-pww0"
server_key: "10394f4ee5d2c39a32e071406a06672d"
server_key_iv: "e78470a625817b77501498d28b8a0ecb"
server_key_version: "encryptionkey@0"
I20260808 04:01:04.321376 22514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:04.337307 22514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:04.341806 22514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:04.342101 22514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:04.343267 22514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:04.344810 22514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:04.344907 22514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:04.345013 22514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:04.345079 22514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:04.377344 22514 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:41725
I20260808 04:01:04.377413 22727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:41725 every 8 connection(s)
I20260808 04:01:04.378895 22514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:01:04.384188 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:04.382130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:38149)
0808 04:01:04.382917 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:04.383183 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":286,"threads_started":1}
W20260808 04:01:04.384969 22729 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:40685 (0 consecutive failures): Network error: Failed to ping master at 127.28.251.254:40685: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
I20260808 04:01:04.385908 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:04.385541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:38253)
0808 04:01:04.385700 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:04.385780 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 04:01:04.386834 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:04.386491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:52683)
0808 04:01:04.386614 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:04.386696 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 04:01:04.387110 22729 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:01:04.389330 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 22514
I20260808 04:01:04.389936 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:38979
--local_ip_for_outbound_sockets=127.28.251.195
--tserver_master_addrs=127.28.251.254:40685
--webserver_port=33147
--webserver_interface=127.28.251.195
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:01:04.757711 22734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:04.758046 22734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:04.758162 22734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:04.765758 22734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:04.765898 22734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:04.766018 22734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:01:04.776283 22734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:38979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=33147
--tserver_master_addrs=127.28.251.254:40685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:04.777782 22734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:04.779173 22734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:04.793334 22743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:04.793254 22747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:04.793280 22744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:04.794909 22734 server_base.cc:1034] running on GCE node
I20260808 04:01:04.795692 22734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:04.796793 22734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:04.798008 22734 hybrid_clock.cc:648] HybridClock initialized: now 1786161664797954 us; error 46 us; skew 500 ppm
I20260808 04:01:04.798367 22734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:04 dist-test-slave-pww0 krb5kdc[20298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161664, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:04.801673 22734 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 04:01:04.803282 22734 webserver.cc:460] Webserver started at http://127.28.251.195:33147/ using document root <none> and password file <none>
I20260808 04:01:04.804035 22734 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:04.804176 22734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:04.810034 22734 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 04:01:04.813580 22760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:04.815169 22734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:01:04.815376 22734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "e96928aff4f647e1846ec372a5d7ed2c"
format_stamp: "Formatted at 2026-08-08 04:00:58 on dist-test-slave-pww0"
server_key: "0ccc6696947ba58e1b382cdc97dfe96d"
server_key_iv: "90442bdeba3c1042471193e4ee54a9ab"
server_key_version: "encryptionkey@0"
I20260808 04:01:04.816560 22734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:04.837262 22734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:04.841658 22734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:04.841945 22734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:04.843065 22734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:04.844794 22734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:04.844902 22734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:04.845024 22734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:04.845098 22734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:04.878499 22734 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:38979
I20260808 04:01:04.878552 22941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:38979 every 8 connection(s)
I20260808 04:01:04.880088 22734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:01:04.885293 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 22734
I20260808 04:01:04.885365 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:04.883176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:53689)
0808 04:01:04.884050 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:04.884342 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":365,"threads_started":1}
W20260808 04:01:04.886049 22944 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:40685 (0 consecutive failures): Network error: Failed to ping master at 127.28.251.254:40685: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
I20260808 04:01:04.886972 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:04.886634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:36705)
0808 04:01:04.886780 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:04.886858 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 04:01:04.888271 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:04.887926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:53391)
0808 04:01:04.888075 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:04.888153 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 04:01:04.888556 22944 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:01:04.893858 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:04.892967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:38443)
0808 04:01:04.893462 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:04.893609 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":234,"threads_started":1}
I20260808 04:01:05.388886 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:05.387831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:54913)
0808 04:01:05.388422 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:05.388609 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":273,"threads_started":1}
I20260808 04:01:05.890293 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:05.889371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:56243)
0808 04:01:05.889887 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:05.890032 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":226,"threads_started":1}
I20260808 04:01:05.895764 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:05.894860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:53831)
0808 04:01:05.895428 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:05.895570 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":309,"threads_started":1}
I20260808 04:01:06.390903 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:06.389951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:43321)
0808 04:01:06.390606 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:06.390717 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":306,"threads_started":1}
I20260808 04:01:06.892172 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:06.891353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:35449)
0808 04:01:06.891893 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:06.891986 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":267,"threads_started":1}
I20260808 04:01:06.897370 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:06.896621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:41503)
0808 04:01:06.897078 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:06.897188 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":221,"threads_started":1}
I20260808 04:01:07.392784 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:07.391892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:39113)
0808 04:01:07.392429 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:07.392551 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":241,"threads_started":1}
I20260808 04:01:07.893869 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:07.893147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:44411)
0808 04:01:07.893593 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:07.893684 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
I20260808 04:01:07.899091 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:07.898305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:57613)
0808 04:01:07.898792 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:07.898886 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":245,"threads_started":1}
I20260808 04:01:08.394632 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:08.393758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:53317)
0808 04:01:08.394250 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:08.394370 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":212,"threads_started":1}
I20260808 04:01:08.895556 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:08.894682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:59657)
0808 04:01:08.895216 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:08.895313 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":294,"threads_started":1}
I20260808 04:01:08.900914 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:08.900122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:37641)
0808 04:01:08.900636 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:08.900727 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":263,"threads_started":1}
I20260808 04:01:09.396739 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:09.395823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:55085)
0808 04:01:09.396434 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:09.396536 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":309,"threads_started":1}
I20260808 04:01:09.897666 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:09.896775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:57009)
0808 04:01:09.897360 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:09.897462 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":277,"threads_started":1}
I20260808 04:01:09.902685 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:09.901890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:32949)
0808 04:01:09.902395 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:09.902496 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":261,"threads_started":1}
I20260808 04:01:10.398568 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:10.397781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:59333)
0808 04:01:10.398278 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:10.398392 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":255,"threads_started":1}
I20260808 04:01:10.899703 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:10.898832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:58385)
0808 04:01:10.899372 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:10.899517 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":277,"threads_started":1}
I20260808 04:01:10.904618 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:10.903685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:33283)
0808 04:01:10.904246 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:10.904342 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":231,"threads_started":1}
I20260808 04:01:11.400695 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:11.399745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:52067)
0808 04:01:11.400354 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:11.400455 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":275,"threads_started":1}
I20260808 04:01:11.901721 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:11.900780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:59817)
0808 04:01:11.901391 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:11.901485 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":286,"threads_started":1}
I20260808 04:01:11.906406 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:11.905647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:51551)
0808 04:01:11.906136 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:11.906238 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":246,"threads_started":1}
I20260808 04:01:12.402660 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:12.401770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:59569)
0808 04:01:12.402377 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:12.402479 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":322,"threads_started":1}
I20260808 04:01:12.903849 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:12.902870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:38623)
0808 04:01:12.903444 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:12.903605 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":297,"threads_started":1}
I20260808 04:01:12.908180 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:12.907342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:45785)
0808 04:01:12.907899 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:12.907989 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":271,"threads_started":1}
I20260808 04:01:13.404707 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:13.403772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:42835)
0808 04:01:13.404423 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:13.404522 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":308,"threads_started":1}
I20260808 04:01:13.905834 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:13.904959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:60777)
0808 04:01:13.905541 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:13.905637 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":291,"threads_started":1}
I20260808 04:01:13.909981 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:13.909212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:40111)
0808 04:01:13.909709 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:13.909808 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":264,"threads_started":1}
I20260808 04:01:14.406705 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:14.405763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:32837)
0808 04:01:14.406413 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:14.406522 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":321,"threads_started":1}
I20260808 04:01:14.907805 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:14.906897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:54115)
0808 04:01:14.907505 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:14.907613 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":298,"threads_started":1}
I20260808 04:01:14.911700 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:14.910910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:34709)
0808 04:01:14.911361 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:14.911499 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":200,"threads_started":1}
I20260808 04:01:15.408888 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:15.407858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:60621)
0808 04:01:15.408519 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:15.408663 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":322,"threads_started":1}
I20260808 04:01:15.909909 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:15.908945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:47781)
0808 04:01:15.909585 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:15.909694 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":413,"threads_started":1}
I20260808 04:01:15.913729 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:15.912822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:54725)
0808 04:01:15.913482 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:15.913565 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":265,"threads_started":1}
I20260808 04:01:16.410940 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:16.410001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:32871)
0808 04:01:16.410636 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:16.410746 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":328,"threads_started":1}
I20260808 04:01:16.911935 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:16.911043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:34973)
0808 04:01:16.911663 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:16.911764 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":300,"threads_started":1}
I20260808 04:01:16.915709 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:16.914744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:41275)
0808 04:01:16.915323 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:16.915441 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":267,"threads_started":1}
I20260808 04:01:17.412952 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:17.412036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:41525)
0808 04:01:17.412599 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:17.412753 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":304,"threads_started":1}
I20260808 04:01:17.913930 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:17.913001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:33159)
0808 04:01:17.913558 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:17.913690 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":289,"threads_started":1}
I20260808 04:01:17.917444 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:17.916645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:40981)
0808 04:01:17.917164 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:17.917252 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":283,"threads_started":1}
I20260808 04:01:18.414817 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:18.413923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:43069)
0808 04:01:18.414460 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:18.414562 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":248,"threads_started":1}
I20260808 04:01:18.915830 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:18.914906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:36829)
0808 04:01:18.915516 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:18.915626 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":306,"threads_started":1}
I20260808 04:01:18.919447 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:18.918610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:53033)
0808 04:01:18.919129 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:18.919236 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":264,"threads_started":1}
I20260808 04:01:19.416975 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:19.416085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.194:40975)
0808 04:01:19.416640 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:19.416744 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":297,"threads_started":1}
I20260808 04:01:19.892341 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 22250
I20260808 04:01:19.915764 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:19.893559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.28.251.193:35607)
0808 04:01:19.894384 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:19.894404 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:19.894455 (+ 51us) server_negotiation.cc:407] Connection header received
0808 04:01:19.896553 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:19.896565 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:19.897030 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:19.897351 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:19.898897 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:19.900229 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:19.901577 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:19.902274 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:19.902431 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:19.908568 (+ 6137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:19.908578 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:19.908597 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:19.908653 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:19.912131 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:19.912927 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:19.912934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:19.912942 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:19.913130 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:19.913579 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:19.913582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:19.913584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:19.914160 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:19.914406 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:19.914985 (+ 579us) server_negotiation.cc:299] Negotiation successful
0808 04:01:19.915416 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":426,"threads_started":1}
I20260808 04:01:19.917444 22503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:19.917825 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:19.916911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.195:47243)
0808 04:01:19.917445 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:19.917574 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":254,"threads_started":1}
I20260808 04:01:20.001802 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:20.000351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40685 (local address 127.28.251.193:45995)
0808 04:01:20.001287 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:20.001508 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":608,"threads_started":1}
I20260808 04:01:20.002153 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 22250
I20260808 04:01:20.031760 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 22514
I20260808 04:01:20.051609 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:20.032734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.28.251.194:41725)
0808 04:01:20.033306 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:20.033321 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:20.033363 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:01:20.034991 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:20.035002 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:20.035445 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:20.035892 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:20.037563 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:20.038872 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:20.040293 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:20.041019 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:20.041232 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:20.044253 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:20.044262 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:20.044280 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:20.044328 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:20.047783 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:20.048620 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:20.048629 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:20.048637 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:20.048853 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:20.049301 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:20.049305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:20.049307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:20.049918 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:20.050240 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:20.050763 (+ 523us) server_negotiation.cc:299] Negotiation successful
0808 04:01:20.051259 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":268,"threads_started":1}
I20260808 04:01:20.053058 22716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:20.138319 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 22514
I20260808 04:01:20.161815 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 22734
I20260808 04:01:20.181301 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:20.162875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.28.251.195:38979)
0808 04:01:20.163616 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:20.163630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:20.163675 (+ 45us) server_negotiation.cc:407] Connection header received
0808 04:01:20.165405 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:20.165415 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:20.165869 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:20.166211 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:20.167797 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:20.169058 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:20.170431 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:20.171138 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:20.171355 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:20.174157 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:20.174166 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:20.174184 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:20.174238 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:20.177545 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:20.178369 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:20.178377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:20.178385 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:20.178580 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:20.178989 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:20.178993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:20.178995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:20.179695 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:20.180023 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:20.180480 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 04:01:20.180965 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":334,"threads_started":1}
I20260808 04:01:20.182796 22935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:20.268412 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 22734
2026-08-08T04:01:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25670 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:01:20 dist-test-slave-pww0 krb5kdc[23037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:01:20 dist-test-slave-pww0 krb5kdc[23037](info): set up 2 sockets
Aug 08 04:01:20 dist-test-slave-pww0 krb5kdc[23037](info): commencing operation
krb5kdc: starting...
W20260808 04:01:21.551322 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:01:21 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:01:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:01:21.897392 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41725
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:41725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:01:22.258046 23063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:22.258410 23063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:22.258566 23063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:22.266249 23063 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:01:22.266386 23063 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:01:22.266434 23063 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:01:22.266516 23063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:01:22.266613 23063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:22.266719 23063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:22.266781 23063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:01:22.266867 23063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:01:22.277587 23063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:41725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:22.279079 23063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:22.280617 23063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:22.293264 23072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:22.293226 23077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:22.293303 23073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:22.294664 23063 server_base.cc:1034] running on GCE node
I20260808 04:01:22.295598 23063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:22.297207 23063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:22.298460 23063 hybrid_clock.cc:648] HybridClock initialized: now 1786161682298386 us; error 49 us; skew 500 ppm
I20260808 04:01:22.298849 23063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:22 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161682, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:22.302852 23063 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:01:22.304639 23063 webserver.cc:460] Webserver started at http://127.28.251.254:46533/ using document root <none> and password file <none>
I20260808 04:01:22.305418 23063 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:22.305531 23063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:22.305868 23063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:01:22.309095 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "7f28a8afb50749e5ae40c4d6e953da93"
format_stamp: "Formatted at 2026-08-08 04:01:22 on dist-test-slave-pww0"
server_key: "26305652e8ee525f850c439151b8e7f7"
server_key_iv: "22e3eaf95821fab825aac237fbb005dc"
server_key_version: "encryptionkey@0"
I20260808 04:01:22.309921 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "7f28a8afb50749e5ae40c4d6e953da93"
format_stamp: "Formatted at 2026-08-08 04:01:22 on dist-test-slave-pww0"
server_key: "26305652e8ee525f850c439151b8e7f7"
server_key_iv: "22e3eaf95821fab825aac237fbb005dc"
server_key_version: "encryptionkey@0"
I20260808 04:01:22.315513 23063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 04:01:22.319316 23088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:22.320819 23063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:01:22.321002 23063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "7f28a8afb50749e5ae40c4d6e953da93"
format_stamp: "Formatted at 2026-08-08 04:01:22 on dist-test-slave-pww0"
server_key: "26305652e8ee525f850c439151b8e7f7"
server_key_iv: "22e3eaf95821fab825aac237fbb005dc"
server_key_version: "encryptionkey@0"
I20260808 04:01:22.321216 23063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:22.359035 23063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:22.363698 23063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:22.363989 23063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:22.383028 23063 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:41725
I20260808 04:01:22.383045 23171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:41725 every 8 connection(s)
I20260808 04:01:22.384479 23063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:01:22.388456 23172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:01:22.392405 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 23063
I20260808 04:01:22.392645 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:01:22.393532 29679 external_mini_cluster.cc:1428] Setting key 0c1a7c78c2c47875af2669bb7b92cddd
I20260808 04:01:22.402631 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Bootstrap starting.
I20260808 04:01:22.406955 23172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Neither blocks nor log segments found. Creating new log.
I20260808 04:01:22.408445 23172 log.cc:824] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:01:22 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:01:22.412180 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: No bootstrap required, opened a new log
I20260808 04:01:22.419209 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:22.395834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.28.251.254:41725)
0808 04:01:22.396672 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:22.396684 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:22.396729 (+ 45us) server_negotiation.cc:407] Connection header received
0808 04:01:22.398389 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:22.398415 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:22.398854 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:22.399203 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:22.401167 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:22.402558 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:22.404633 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:22.405448 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:22.405599 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:22.410996 (+ 5397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:22.411004 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:22.411021 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:22.411066 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:22.414456 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:22.415502 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:22.415509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:22.415517 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:22.415708 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:22.416151 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:22.416154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:22.416156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:22.416823 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:22.417077 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:22.417584 (+ 507us) server_negotiation.cc:299] Negotiation successful
0808 04:01:22.418017 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":348,"threads_started":1}
I20260808 04:01:22.421370 23172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:01:22.421787 23172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:01:22.421901 23172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f28a8afb50749e5ae40c4d6e953da93, State: Initialized, Role: FOLLOWER
I20260808 04:01:22.422570 23172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:01:22.422811 23172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:01:22.422943 23172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:01:22.423099 23172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:01:22.425750 23172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:01:22.426352 23172 leader_election.cc:302] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f28a8afb50749e5ae40c4d6e953da93; no voters:
I20260808 04:01:22.426935 23172 leader_election.cc:288] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:01:22.427079 23184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:01:22.427796 23184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 1 LEADER]: Becoming Leader. State: Replica: 7f28a8afb50749e5ae40c4d6e953da93, State: Running, Role: LEADER
I20260808 04:01:22.428339 23184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:01:22.429004 23172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:01:22.431124 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f28a8afb50749e5ae40c4d6e953da93. Latest consensus state: current_term: 1 leader_uuid: "7f28a8afb50749e5ae40c4d6e953da93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } } }
I20260808 04:01:22.431586 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: This master's current role is: LEADER
I20260808 04:01:22.433034 23185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f28a8afb50749e5ae40c4d6e953da93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } } }
I20260808 04:01:22.433449 23185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: This master's current role is: LEADER
I20260808 04:01:22.433682 23191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:01:22.439258 23191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:01:22.449783 23191 catalog_manager.cc:1347] Generated new cluster ID: 022c774db10a4670bcef7985358315ba
I20260808 04:01:22.449916 23191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:01:22.485534 23191 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:01:22.487198 23191 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:01:22.505195 23191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Generated new TSK 0
I20260808 04:01:22.506347 23191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 04:01:22.591579 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41725
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:01:22.969831 23219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:22.970232 23219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:22.970412 23219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:22.978739 23219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:22.978875 23219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:22.979040 23219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:01:22.989457 23219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:22.990921 23219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:22.992333 23219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:23.005924 23226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:23.005977 23230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:23.005879 23227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:23.006841 23219 server_base.cc:1034] running on GCE node
I20260808 04:01:23.007562 23219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:23.008587 23219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:23.009788 23219 hybrid_clock.cc:648] HybridClock initialized: now 1786161683009739 us; error 37 us; skew 500 ppm
I20260808 04:01:23.010118 23219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:23 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161683, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:23.013511 23219 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 04:01:23.015148 23219 webserver.cc:460] Webserver started at http://127.28.251.193:34419/ using document root <none> and password file <none>
I20260808 04:01:23.015962 23219 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:23.016088 23219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:23.016412 23219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:01:23.019619 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "e161f3da09a342d7afab36246bccebbd"
format_stamp: "Formatted at 2026-08-08 04:01:23 on dist-test-slave-pww0"
server_key: "b377483e6428c897ba3a839dc20c4546"
server_key_iv: "3439b9ad267f2b4b2a859f2f336ba0b1"
server_key_version: "encryptionkey@0"
I20260808 04:01:23.020440 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "e161f3da09a342d7afab36246bccebbd"
format_stamp: "Formatted at 2026-08-08 04:01:23 on dist-test-slave-pww0"
server_key: "b377483e6428c897ba3a839dc20c4546"
server_key_iv: "3439b9ad267f2b4b2a859f2f336ba0b1"
server_key_version: "encryptionkey@0"
I20260808 04:01:23.026126 23219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:01:23.030161 23241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:23.031854 23219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:01:23.032044 23219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "e161f3da09a342d7afab36246bccebbd"
format_stamp: "Formatted at 2026-08-08 04:01:23 on dist-test-slave-pww0"
server_key: "b377483e6428c897ba3a839dc20c4546"
server_key_iv: "3439b9ad267f2b4b2a859f2f336ba0b1"
server_key_version: "encryptionkey@0"
I20260808 04:01:23.032260 23219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:23.050073 23219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:23.054489 23219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:23.054767 23219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:23.055794 23219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:23.057264 23219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:23.057340 23219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:23.057439 23219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:23.057483 23219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:23.092075 23219 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:44809
I20260808 04:01:23.092109 23446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:44809 every 8 connection(s)
I20260808 04:01:23.093479 23219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:01:23.097458 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 23219
I20260808 04:01:23.097708 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 04:01:23.098608 29679 external_mini_cluster.cc:1428] Setting key 995d62144e02e2bd9010a9b7e8266f6c
Aug 08 04:01:23 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161683, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:01:23.116312 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:23.096250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:34371 (local address 127.28.251.254:41725)
0808 04:01:23.096762 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:23.096766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:23.098057 (+ 1291us) server_negotiation.cc:407] Connection header received
0808 04:01:23.100557 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:23.100562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:23.100642 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:23.100742 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:23.102582 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:23.103708 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:23.105425 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:23.105982 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:23.106116 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:23.110027 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:23.110034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:23.110037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:23.110075 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:23.112753 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:23.113748 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:23.113754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:23.113757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:23.113909 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:23.114393 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:23.114397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:23.114400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:23.114708 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:23.114835 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:23.115751 (+ 916us) server_negotiation.cc:299] Negotiation successful
0808 04:01:23.115976 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":244,"threads_started":1}
I20260808 04:01:23.117497 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:23.096897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:34371)
0808 04:01:23.097813 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:23.097897 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:23.100214 (+ 2317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:23.101072 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:23.101083 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:23.101677 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:01:23.102364 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:23.102386 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:23.103932 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:23.103937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:23.105258 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:23.105272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:23.106214 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:23.106219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:23.106440 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:23.107137 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:01:23.107164 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:01:23.109735 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:01:23.112920 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:23.112930 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:23.112951 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:23.113598 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:23.114023 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:23.114027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:23.114030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:23.114295 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:23.114838 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:01:23.114847 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:01:23.115312 (+ 465us) client_negotiation.cc:769] Sending connection context
0808 04:01:23.115669 (+ 357us) client_negotiation.cc:240] Negotiation successful
0808 04:01:23.116233 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":320,"threads_started":1}
I20260808 04:01:23.119367 23447 heartbeater.cc:347] Connected to a master server at 127.28.251.254:41725
I20260808 04:01:23.119712 23447 heartbeater.cc:464] Registering TS with master...
I20260808 04:01:23.120505 23447 heartbeater.cc:511] Master 127.28.251.254:41725 requested a full tablet report, sending...
I20260808 04:01:23.122875 23111 ts_manager.cc:194] Registered new tserver with Master: e161f3da09a342d7afab36246bccebbd (127.28.251.193:44809)
I20260808 04:01:23.125051 23111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:34371
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 04:01:23.176993 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41725
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:01:23.545475 23458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:23.545826 23458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:23.545933 23458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:23.553442 23458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:23.553571 23458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:23.553689 23458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:01:23.563886 23458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:23.565240 23458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:23.566606 23458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:23.580487 23463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:23.580471 23467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:23.580636 23464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:23.581403 23458 server_base.cc:1034] running on GCE node
I20260808 04:01:23.582072 23458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:23.583070 23458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:23.584295 23458 hybrid_clock.cc:648] HybridClock initialized: now 1786161683584229 us; error 50 us; skew 500 ppm
I20260808 04:01:23.584642 23458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:23 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161683, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:23.587947 23458 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 04:01:23.589452 23458 webserver.cc:460] Webserver started at http://127.28.251.194:43591/ using document root <none> and password file <none>
I20260808 04:01:23.590163 23458 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:23.590274 23458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:23.590572 23458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:01:23.594570 23458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "c2956e6c23b842d8be00d1e2c200b612"
format_stamp: "Formatted at 2026-08-08 04:01:23 on dist-test-slave-pww0"
server_key: "97e112a3e729ea4a358ebadcb5f5f125"
server_key_iv: "2cff378ded54b402c67156dccd93677e"
server_key_version: "encryptionkey@0"
I20260808 04:01:23.595400 23458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "c2956e6c23b842d8be00d1e2c200b612"
format_stamp: "Formatted at 2026-08-08 04:01:23 on dist-test-slave-pww0"
server_key: "97e112a3e729ea4a358ebadcb5f5f125"
server_key_iv: "2cff378ded54b402c67156dccd93677e"
server_key_version: "encryptionkey@0"
I20260808 04:01:23.601164 23458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 04:01:23.605064 23482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:23.606520 23458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 04:01:23.606701 23458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "c2956e6c23b842d8be00d1e2c200b612"
format_stamp: "Formatted at 2026-08-08 04:01:23 on dist-test-slave-pww0"
server_key: "97e112a3e729ea4a358ebadcb5f5f125"
server_key_iv: "2cff378ded54b402c67156dccd93677e"
server_key_version: "encryptionkey@0"
I20260808 04:01:23.606930 23458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:23.636415 23458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:23.640957 23458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:23.641245 23458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:23.642324 23458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:23.643890 23458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:23.643966 23458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:23.644088 23458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:23.644140 23458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:23.676026 23458 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:42723
I20260808 04:01:23.676097 23664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:42723 every 8 connection(s)
I20260808 04:01:23.677341 23458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:01:23.682740 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 23458
I20260808 04:01:23.682958 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 04:01:23.683578 29679 external_mini_cluster.cc:1428] Setting key bdcb3889cd03c0601fa490f69fdfdb0f
Aug 08 04:01:23 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161683, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:01:23.699399 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:23.680025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:33073 (local address 127.28.251.254:41725)
0808 04:01:23.680491 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:23.680495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:23.681481 (+ 986us) server_negotiation.cc:407] Connection header received
0808 04:01:23.683840 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:23.683847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:23.683934 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:23.684033 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:23.685884 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:23.686638 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:23.688910 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:23.689415 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:23.689513 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:23.693657 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:23.693662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:23.693666 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:23.693702 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:23.696251 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:23.697125 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:23.697129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:23.697132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:23.697257 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:23.697722 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:23.697725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:23.697727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:23.698011 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:23.698126 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:23.698936 (+ 810us) server_negotiation.cc:299] Negotiation successful
0808 04:01:23.699108 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260808 04:01:23.700819 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:23.680543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:33073)
0808 04:01:23.681285 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:23.681350 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:23.683496 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:23.684324 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:23.684333 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:23.684985 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:01:23.685695 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:23.685722 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:23.686807 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:23.686812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:23.688762 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:23.688778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:23.689652 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:23.689659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:23.689922 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:23.690683 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:01:23.690713 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:01:23.693401 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:01:23.696417 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:23.696426 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:23.696444 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:23.697010 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:23.697352 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:23.697356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:23.697358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:23.697603 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:23.698132 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:01:23.698139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:01:23.698579 (+ 440us) client_negotiation.cc:769] Sending connection context
0808 04:01:23.698871 (+ 292us) client_negotiation.cc:240] Negotiation successful
0808 04:01:23.699418 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":288,"threads_started":1}
I20260808 04:01:23.702595 23666 heartbeater.cc:347] Connected to a master server at 127.28.251.254:41725
I20260808 04:01:23.702991 23666 heartbeater.cc:464] Registering TS with master...
I20260808 04:01:23.703812 23666 heartbeater.cc:511] Master 127.28.251.254:41725 requested a full tablet report, sending...
I20260808 04:01:23.705350 23111 ts_manager.cc:194] Registered new tserver with Master: c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723)
I20260808 04:01:23.706218 23111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:33073
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 04:01:23.756840 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41725
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:01:24.118132 23674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:24.118470 23674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:24.118604 23674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:24.126618 23674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:24.126756 23674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:24.126895 23674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:01:24.129812 23447 heartbeater.cc:503] Master 127.28.251.254:41725 was elected leader, sending a full tablet report...
I20260808 04:01:24.137933 23674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:24.139389 23674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:24.140790 23674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:24.154660 23681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:24.154645 23680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:24.154906 23684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:24.155575 23674 server_base.cc:1034] running on GCE node
I20260808 04:01:24.156292 23674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:24.157449 23674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:24.158720 23674 hybrid_clock.cc:648] HybridClock initialized: now 1786161684158658 us; error 42 us; skew 500 ppm
I20260808 04:01:24.159097 23674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:24 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161684, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:24.162714 23674 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 04:01:24.164562 23674 webserver.cc:460] Webserver started at http://127.28.251.195:45759/ using document root <none> and password file <none>
I20260808 04:01:24.165355 23674 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:24.165498 23674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:24.165824 23674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:01:24.169126 23674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "03148b5891fe444dac3625203b23adb9"
format_stamp: "Formatted at 2026-08-08 04:01:24 on dist-test-slave-pww0"
server_key: "50eba8b20b7ca7e8c4233dbe2b2123c7"
server_key_iv: "b179bd56d6ee544c7ae2c5e83563fa65"
server_key_version: "encryptionkey@0"
I20260808 04:01:24.169983 23674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "03148b5891fe444dac3625203b23adb9"
format_stamp: "Formatted at 2026-08-08 04:01:24 on dist-test-slave-pww0"
server_key: "50eba8b20b7ca7e8c4233dbe2b2123c7"
server_key_iv: "b179bd56d6ee544c7ae2c5e83563fa65"
server_key_version: "encryptionkey@0"
I20260808 04:01:24.176014 23674 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 04:01:24.180058 23693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:24.181586 23674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:01:24.181787 23674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "03148b5891fe444dac3625203b23adb9"
format_stamp: "Formatted at 2026-08-08 04:01:24 on dist-test-slave-pww0"
server_key: "50eba8b20b7ca7e8c4233dbe2b2123c7"
server_key_iv: "b179bd56d6ee544c7ae2c5e83563fa65"
server_key_version: "encryptionkey@0"
I20260808 04:01:24.182015 23674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:24.204643 23674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:24.209535 23674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:24.209851 23674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:24.211023 23674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:24.212620 23674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:24.212704 23674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:24.212821 23674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:24.212869 23674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:24.245425 23674 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:36103
I20260808 04:01:24.245478 23879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:36103 every 8 connection(s)
I20260808 04:01:24.246858 23674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:01:24.251559 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 23674
I20260808 04:01:24.251760 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 04:01:24.252233 29679 external_mini_cluster.cc:1428] Setting key 7ac1829821568dc2ee091794010b09ed
Aug 08 04:01:24 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161684, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:01:24.268308 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.249745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:37795 (local address 127.28.251.254:41725)
0808 04:01:24.250260 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.250267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.251402 (+ 1135us) server_negotiation.cc:407] Connection header received
0808 04:01:24.253605 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.253611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.253702 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.253799 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:24.255495 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.256250 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.257874 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.258294 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.258409 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.262184 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:24.262190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:24.262193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:24.262239 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:24.264940 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:24.265757 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:24.265762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:24.265764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:24.265904 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:24.266374 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:24.266378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:24.266380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:24.266688 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:24.266833 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.267636 (+ 803us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.267885 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":218,"threads_started":1}
I20260808 04:01:24.269394 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.250338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:37795)
0808 04:01:24.251166 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:24.251251 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:24.253317 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:24.254080 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:24.254089 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:24.254672 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:01:24.255291 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.255311 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.256411 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.256416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.257720 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.257734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.258587 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.258595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.258824 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.259573 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:01:24.259610 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:01:24.261935 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:01:24.265108 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:24.265119 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:24.265138 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:24.265618 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:24.266023 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:24.266029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:24.266033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:24.266254 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:24.266831 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:01:24.266840 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:01:24.267220 (+ 380us) client_negotiation.cc:769] Sending connection context
0808 04:01:24.267529 (+ 309us) client_negotiation.cc:240] Negotiation successful
0808 04:01:24.268155 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":319,"threads_started":1}
I20260808 04:01:24.270885 23880 heartbeater.cc:347] Connected to a master server at 127.28.251.254:41725
I20260808 04:01:24.271245 23880 heartbeater.cc:464] Registering TS with master...
I20260808 04:01:24.272074 23880 heartbeater.cc:511] Master 127.28.251.254:41725 requested a full tablet report, sending...
I20260808 04:01:24.273650 23111 ts_manager.cc:194] Registered new tserver with Master: 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103)
I20260808 04:01:24.274497 23111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:37795
I20260808 04:01:24.280994 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:01:24.299834 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.285823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.28.251.254:41725)
0808 04:01:24.286060 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.286067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.286359 (+ 292us) server_negotiation.cc:407] Connection header received
0808 04:01:24.286552 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.286557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.286711 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.286826 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:24.288356 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.289106 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.290616 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.291113 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.291233 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.293928 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:24.293933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:24.293936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:24.293975 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:24.296683 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:24.297504 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:24.297507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:24.297510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:24.297646 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:24.298083 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:24.298087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:24.298089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:24.298452 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:24.298685 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.299181 (+ 496us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.299490 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:01:24.304389 23111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:01:24.311583 23111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:01:24.336934 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.325051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.28.251.195:36103)
0808 04:01:24.326212 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.326221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.326814 (+ 593us) server_negotiation.cc:407] Connection header received
0808 04:01:24.327331 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.327340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.327678 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.327912 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:24.329340 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.331039 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.333985 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.335393 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.335643 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.335862 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.336217 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.336476 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":840,"threads_started":1}
I20260808 04:01:24.337020 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.327574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.28.251.194:42723)
0808 04:01:24.328604 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.328614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.329259 (+ 645us) server_negotiation.cc:407] Connection header received
0808 04:01:24.329853 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.329861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.330077 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.330199 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:24.330885 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.332334 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.334671 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.335860 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.336028 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.336195 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.336467 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.336648 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":720,"threads_started":1}
I20260808 04:01:24.337949 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.326027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.28.251.193:44809)
0808 04:01:24.326814 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.326822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.327536 (+ 714us) server_negotiation.cc:407] Connection header received
0808 04:01:24.327640 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.328812 (+ 1172us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.329073 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.329281 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:24.329935 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.331521 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.333810 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.335392 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.337188 (+ 1796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.337376 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.337449 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.337626 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":448,"threads_started":1}
I20260808 04:01:24.338081 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.327113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:42723 (local address 127.0.0.1:55472)
0808 04:01:24.329150 (+ 2037us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:24.329172 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:24.329387 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:24.330420 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:24.330425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:24.330450 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:24.330734 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.330742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.332518 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.332522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.334520 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.334532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.336075 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.336081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.336243 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.336276 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:01:24.337487 (+ 1211us) client_negotiation.cc:240] Negotiation successful
0808 04:01:24.337626 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1925,"mutex_wait_us":224,"thread_start_us":1549,"threads_started":1}
I20260808 04:01:24.338032 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.324747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:36103 (local address 127.0.0.1:34220)
0808 04:01:24.326634 (+ 1887us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:24.326694 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:24.326992 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:24.328572 (+ 1580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:24.328579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:24.328620 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:24.329109 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.329125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.331190 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.331195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.333809 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.333824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.335642 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.335649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.335915 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.335982 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 04:01:24.337577 (+ 1595us) client_negotiation.cc:240] Negotiation successful
0808 04:01:24.337690 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1631,"mutex_wait_us":53,"thread_start_us":1479,"threads_started":1}
I20260808 04:01:24.338348 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.325383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.0.0.1:37908)
0808 04:01:24.327263 (+ 1880us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:24.327287 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:24.327402 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:24.329281 (+ 1879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:24.329286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:24.329321 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:24.329765 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.329779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.331702 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.331707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.333647 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.333660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.336396 (+ 2736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.336403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.336580 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.336613 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:01:24.336685 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:01:24.336903 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1753,"mutex_wait_us":23,"thread_start_us":1562,"threads_started":1}
I20260808 04:01:24.342595 23772 tablet_service.cc:1467] Processing CreateTablet for tablet f02aa07e99f948d1b0ef03f52b070512 (DEFAULT_TABLE table=test-table [id=6d9aeb0a4dbc498aa3cc4c565ea62dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:01:24.342595 23329 tablet_service.cc:1467] Processing CreateTablet for tablet f02aa07e99f948d1b0ef03f52b070512 (DEFAULT_TABLE table=test-table [id=6d9aeb0a4dbc498aa3cc4c565ea62dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:01:24.344276 23329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02aa07e99f948d1b0ef03f52b070512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:01:24.344276 23772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02aa07e99f948d1b0ef03f52b070512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:01:24.345717 23561 tablet_service.cc:1467] Processing CreateTablet for tablet f02aa07e99f948d1b0ef03f52b070512 (DEFAULT_TABLE table=test-table [id=6d9aeb0a4dbc498aa3cc4c565ea62dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:01:24.346975 23561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02aa07e99f948d1b0ef03f52b070512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:01:24.358829 23903 tablet_bootstrap.cc:492] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: Bootstrap starting.
I20260808 04:01:24.359890 23904 tablet_bootstrap.cc:492] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: Bootstrap starting.
I20260808 04:01:24.363847 23904 tablet_bootstrap.cc:654] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: Neither blocks nor log segments found. Creating new log.
I20260808 04:01:24.363842 23903 tablet_bootstrap.cc:654] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: Neither blocks nor log segments found. Creating new log.
I20260808 04:01:24.364259 23905 tablet_bootstrap.cc:492] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: Bootstrap starting.
I20260808 04:01:24.365284 23904 log.cc:824] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:24.366616 23903 log.cc:824] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:24.368165 23905 tablet_bootstrap.cc:654] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: Neither blocks nor log segments found. Creating new log.
I20260808 04:01:24.369338 23905 log.cc:824] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:24.370370 23904 tablet_bootstrap.cc:492] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: No bootstrap required, opened a new log
I20260808 04:01:24.370370 23903 tablet_bootstrap.cc:492] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: No bootstrap required, opened a new log
I20260808 04:01:24.370698 23904 ts_tablet_manager.cc:1397] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260808 04:01:24.370698 23903 ts_tablet_manager.cc:1397] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260808 04:01:24.372242 23905 tablet_bootstrap.cc:492] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: No bootstrap required, opened a new log
I20260808 04:01:24.372512 23905 ts_tablet_manager.cc:1397] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 04:01:24.380190 23905 raft_consensus.cc:348] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } }
I20260808 04:01:24.380637 23905 raft_consensus.cc:374] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:01:24.380740 23905 raft_consensus.cc:729] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03148b5891fe444dac3625203b23adb9, State: Initialized, Role: FOLLOWER
I20260808 04:01:24.381362 23905 consensus_queue.cc:260] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } }
I20260808 04:01:24.381855 23904 raft_consensus.cc:348] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } }
I20260808 04:01:24.381857 23903 raft_consensus.cc:348] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } }
I20260808 04:01:24.382339 23903 raft_consensus.cc:374] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:01:24.382344 23904 raft_consensus.cc:374] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:01:24.382452 23903 raft_consensus.cc:729] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2956e6c23b842d8be00d1e2c200b612, State: Initialized, Role: FOLLOWER
I20260808 04:01:24.382452 23904 raft_consensus.cc:729] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e161f3da09a342d7afab36246bccebbd, State: Initialized, Role: FOLLOWER
I20260808 04:01:24.382704 23880 heartbeater.cc:503] Master 127.28.251.254:41725 was elected leader, sending a full tablet report...
I20260808 04:01:24.383199 23904 consensus_queue.cc:260] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } }
I20260808 04:01:24.383235 23903 consensus_queue.cc:260] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } }
I20260808 04:01:24.388607 23904 ts_tablet_manager.cc:1428] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: Time spent starting tablet: real 0.018s user 0.017s sys 0.001s
I20260808 04:01:24.388664 23905 ts_tablet_manager.cc:1428] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: Time spent starting tablet: real 0.016s user 0.011s sys 0.000s
I20260808 04:01:24.393293 23666 heartbeater.cc:503] Master 127.28.251.254:41725 was elected leader, sending a full tablet report...
I20260808 04:01:24.394222 23903 ts_tablet_manager.cc:1428] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: Time spent starting tablet: real 0.023s user 0.018s sys 0.004s
W20260808 04:01:24.429375 23667 tablet.cc:2369] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:01:24.487593 23916 raft_consensus.cc:482] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:01:24.487942 23916 raft_consensus.cc:504] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } }
I20260808 04:01:24.489719 23916 leader_election.cc:288] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103), e161f3da09a342d7afab36246bccebbd (127.28.251.193:44809)
I20260808 04:01:24.496764 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.490314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:43521 (local address 127.28.251.193:44809)
0808 04:01:24.490557 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.490563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.491287 (+ 724us) server_negotiation.cc:407] Connection header received
0808 04:01:24.491509 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.491515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.491576 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.491651 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:24.492335 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.493063 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.494985 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.495831 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.495991 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.496075 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.496376 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.496516 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260808 04:01:24.497054 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.490485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:43239 (local address 127.28.251.195:36103)
0808 04:01:24.490703 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.490708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.491349 (+ 641us) server_negotiation.cc:407] Connection header received
0808 04:01:24.491550 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.491554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.491615 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.491691 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:24.492506 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.493254 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.495174 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.496095 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.496253 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.496331 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.496651 (+ 320us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.496788 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:01:24.497179 23924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.490313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:36103 (local address 127.28.251.194:43239)
0808 04:01:24.491230 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:24.491245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:24.491386 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:24.491720 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:24.491724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:24.491749 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:24.492333 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.492352 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.493411 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.493416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.495000 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.495017 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.496319 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.496324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.496494 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.496520 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:01:24.496604 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:01:24.496740 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":21,"thread_start_us":563,"threads_started":1}
I20260808 04:01:24.497467 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.490268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.28.251.194:43521)
0808 04:01:24.491159 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:24.491179 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:24.491348 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:24.491690 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:24.491694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:24.491718 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:24.492181 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.492194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.493199 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.493204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.494814 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.494828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.496064 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.496070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.496209 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.496236 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:01:24.496336 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:01:24.496488 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":2,"spinlock_wait_cycles":18176,"thread_start_us":488,"threads_started":1}
I20260808 04:01:24.497454 23915 raft_consensus.cc:482] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:01:24.497833 23915 raft_consensus.cc:504] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } }
W20260808 04:01:24.498658 23881 tablet.cc:2369] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:01:24.498606 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02aa07e99f948d1b0ef03f52b070512" candidate_uuid: "c2956e6c23b842d8be00d1e2c200b612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e161f3da09a342d7afab36246bccebbd" is_pre_election: true
I20260808 04:01:24.498600 23799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02aa07e99f948d1b0ef03f52b070512" candidate_uuid: "c2956e6c23b842d8be00d1e2c200b612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03148b5891fe444dac3625203b23adb9" is_pre_election: true
I20260808 04:01:24.499187 23364 raft_consensus.cc:2454] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2956e6c23b842d8be00d1e2c200b612 in term 0.
I20260808 04:01:24.499153 23799 raft_consensus.cc:2454] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2956e6c23b842d8be00d1e2c200b612 in term 0.
I20260808 04:01:24.500011 23483 leader_election.cc:302] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03148b5891fe444dac3625203b23adb9, c2956e6c23b842d8be00d1e2c200b612, e161f3da09a342d7afab36246bccebbd; no voters:
I20260808 04:01:24.500494 23915 leader_election.cc:288] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723), e161f3da09a342d7afab36246bccebbd (127.28.251.193:44809)
I20260808 04:01:24.500744 23916 raft_consensus.cc:2781] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:01:24.500937 23916 raft_consensus.cc:482] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:01:24.501060 23916 raft_consensus.cc:3037] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:01:24.504622 23916 raft_consensus.cc:504] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } }
I20260808 04:01:24.506081 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02aa07e99f948d1b0ef03f52b070512" candidate_uuid: "c2956e6c23b842d8be00d1e2c200b612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e161f3da09a342d7afab36246bccebbd"
I20260808 04:01:24.506080 23799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02aa07e99f948d1b0ef03f52b070512" candidate_uuid: "c2956e6c23b842d8be00d1e2c200b612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03148b5891fe444dac3625203b23adb9"
I20260808 04:01:24.506330 23799 raft_consensus.cc:3037] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:01:24.506336 23364 raft_consensus.cc:3037] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:01:24.507763 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.500265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:42723 (local address 127.28.251.195:58595)
0808 04:01:24.500471 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:24.500494 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:24.500616 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:24.501136 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:24.501141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:24.501170 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:24.501382 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.501393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.502850 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.502856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.505020 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.505042 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.506917 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.506923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.507128 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.507169 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 04:01:24.507267 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 04:01:24.507423 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 04:01:24.508570 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.500527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:58595 (local address 127.28.251.194:42723)
0808 04:01:24.500711 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.500718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.500741 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:01:24.500802 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.500807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.500884 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.500987 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:24.501549 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.502674 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.505217 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.506466 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.506631 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.506707 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.508174 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.508323 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:01:24.509020 23916 leader_election.cc:288] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [CANDIDATE]: Term 1 election: Requested vote from peers 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103), e161f3da09a342d7afab36246bccebbd (127.28.251.193:44809)
I20260808 04:01:24.509527 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.500851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:35211 (local address 127.28.251.193:44809)
0808 04:01:24.501655 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.501662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.502211 (+ 549us) server_negotiation.cc:407] Connection header received
0808 04:01:24.502356 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.502361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.502425 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.502496 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:24.503183 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.504373 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.506875 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.508273 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.508454 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.508524 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.509093 (+ 569us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.509236 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668}
I20260808 04:01:24.509913 23799 raft_consensus.cc:2454] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2956e6c23b842d8be00d1e2c200b612 in term 1.
I20260808 04:01:24.510324 23925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.500322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.28.251.195:35211)
0808 04:01:24.502105 (+ 1783us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:24.502131 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:24.502244 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:24.502524 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:24.502529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:24.502553 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:24.502977 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.502989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.504589 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.504595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.506672 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:24.506692 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.508726 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.508732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:24.508902 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.508936 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:01:24.509712 (+ 776us) client_negotiation.cc:240] Negotiation successful
0808 04:01:24.509863 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1686,"mutex_wait_us":595,"thread_start_us":886,"threads_started":1}
I20260808 04:01:24.510152 23591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02aa07e99f948d1b0ef03f52b070512" candidate_uuid: "03148b5891fe444dac3625203b23adb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2956e6c23b842d8be00d1e2c200b612" is_pre_election: true
I20260808 04:01:24.510488 23484 leader_election.cc:302] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03148b5891fe444dac3625203b23adb9, c2956e6c23b842d8be00d1e2c200b612; no voters:
I20260808 04:01:24.510731 23591 raft_consensus.cc:2379] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 03148b5891fe444dac3625203b23adb9 in current term 1: Already voted for candidate c2956e6c23b842d8be00d1e2c200b612 in this term.
I20260808 04:01:24.511044 23916 raft_consensus.cc:2781] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:01:24.511497 23364 raft_consensus.cc:2454] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2956e6c23b842d8be00d1e2c200b612 in term 1.
I20260808 04:01:24.511541 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02aa07e99f948d1b0ef03f52b070512" candidate_uuid: "03148b5891fe444dac3625203b23adb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e161f3da09a342d7afab36246bccebbd" is_pre_election: true
I20260808 04:01:24.512004 23363 raft_consensus.cc:2379] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 03148b5891fe444dac3625203b23adb9 in current term 1: Already voted for candidate c2956e6c23b842d8be00d1e2c200b612 in this term.
I20260808 04:01:24.512108 23916 raft_consensus.cc:686] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 1 LEADER]: Becoming Leader. State: Replica: c2956e6c23b842d8be00d1e2c200b612, State: Running, Role: LEADER
I20260808 04:01:24.512698 23694 leader_election.cc:302] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03148b5891fe444dac3625203b23adb9; no voters: c2956e6c23b842d8be00d1e2c200b612, e161f3da09a342d7afab36246bccebbd
I20260808 04:01:24.512652 23916 consensus_queue.cc:237] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } }
I20260808 04:01:24.513070 23915 raft_consensus.cc:2726] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 04:01:24.517807 23108 catalog_manager.cc:5495] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194). New cstate: current_term: 1 leader_uuid: "c2956e6c23b842d8be00d1e2c200b612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } health_report { overall_health: UNKNOWN } } }
I20260808 04:01:24.579663 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.574229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.28.251.194:42723)
0808 04:01:24.574459 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.574464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.574484 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:01:24.574658 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.574662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.574757 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.574885 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:01:24.575270 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.576045 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.577596 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.577982 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.578078 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.578541 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:01:24.579045 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:01:24.579170 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.579264 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.579336 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 04:01:24.588361 23363 raft_consensus.cc:1261] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Refusing update from remote peer c2956e6c23b842d8be00d1e2c200b612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:01:24.588368 23799 raft_consensus.cc:1261] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 1 FOLLOWER]: Refusing update from remote peer c2956e6c23b842d8be00d1e2c200b612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:01:24.589264 23916 consensus_queue.cc:1035] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:01:24.589588 23926 consensus_queue.cc:1035] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:01:24.596045 23448 tablet.cc:2369] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:01:24.602656 23933 mvcc.cc:205] Tried to move back new op lower bound from 7316118260063653888 to 7316118259767898112. Current Snapshot: MvccSnapshot[applied={T|T < 7316118260063653888}]
I20260808 04:01:24.619673 23108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:01:24.620014 23108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:01:24.624320 23108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Sending DeleteTablet for 3 replicas of tablet f02aa07e99f948d1b0ef03f52b070512
I20260808 04:01:24.625658 23772 tablet_service.cc:1514] Processing DeleteTablet for tablet f02aa07e99f948d1b0ef03f52b070512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:01:24 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:34220
I20260808 04:01:24.625826 23329 tablet_service.cc:1514] Processing DeleteTablet for tablet f02aa07e99f948d1b0ef03f52b070512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:01:24 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:37908
I20260808 04:01:24.626156 23561 tablet_service.cc:1514] Processing DeleteTablet for tablet f02aa07e99f948d1b0ef03f52b070512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:01:24 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:55472
I20260808 04:01:24.626992 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 23219
I20260808 04:01:24.640717 23956 tablet_replica.cc:330] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: stopping tablet replica
I20260808 04:01:24.640972 23954 tablet_replica.cc:330] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: stopping tablet replica
I20260808 04:01:24.641423 23954 raft_consensus.cc:2229] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:01:24.642019 23952 tablet_replica.cc:330] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: stopping tablet replica
I20260808 04:01:24.642023 23954 raft_consensus.cc:2258] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:01:24.642711 23956 raft_consensus.cc:2229] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:01:24.643496 23956 raft_consensus.cc:2258] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:01:24.644215 23952 raft_consensus.cc:2229] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:01:24.644269 23954 ts_tablet_manager.cc:1905] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:01:24.644757 23952 raft_consensus.cc:2258] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:01:24.647183 23956 ts_tablet_manager.cc:1905] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:01:24.650185 23952 ts_tablet_manager.cc:1905] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:01:24.650983 23954 ts_tablet_manager.cc:1918] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:01:24.651269 23954 log.cc:1196] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/f02aa07e99f948d1b0ef03f52b070512
I20260808 04:01:24.651913 23954 ts_tablet_manager.cc:1939] T f02aa07e99f948d1b0ef03f52b070512 P e161f3da09a342d7afab36246bccebbd: Deleting consensus metadata
I20260808 04:01:24.654104 23089 catalog_manager.cc:4841] TS e161f3da09a342d7afab36246bccebbd (127.28.251.193:44809): tablet f02aa07e99f948d1b0ef03f52b070512 (table test-table [id=6d9aeb0a4dbc498aa3cc4c565ea62dac]) successfully deleted
Aug 08 04:01:24 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 04:01:24.658499 23952 ts_tablet_manager.cc:1918] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:01:24.658812 23952 log.cc:1196] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/f02aa07e99f948d1b0ef03f52b070512
I20260808 04:01:24.659503 23952 ts_tablet_manager.cc:1939] T f02aa07e99f948d1b0ef03f52b070512 P 03148b5891fe444dac3625203b23adb9: Deleting consensus metadata
I20260808 04:01:24.661680 23090 catalog_manager.cc:4841] TS 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103): tablet f02aa07e99f948d1b0ef03f52b070512 (table test-table [id=6d9aeb0a4dbc498aa3cc4c565ea62dac]) successfully deleted
I20260808 04:01:24.662365 23956 ts_tablet_manager.cc:1918] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:01:24.662602 23956 log.cc:1196] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/f02aa07e99f948d1b0ef03f52b070512
I20260808 04:01:24.663209 23956 ts_tablet_manager.cc:1939] T f02aa07e99f948d1b0ef03f52b070512 P c2956e6c23b842d8be00d1e2c200b612: Deleting consensus metadata
I20260808 04:01:24.663722 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.628246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.28.251.193:44809)
0808 04:01:24.628439 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.628445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.634906 (+ 6461us) server_negotiation.cc:407] Connection header received
0808 04:01:24.635242 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.635249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.635426 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.635615 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:24.638252 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.639598 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.641856 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.642561 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.642698 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.655760 (+ 13062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:24.655776 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:24.655795 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:24.655845 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:24.659755 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:24.660779 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:24.660790 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:24.660793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:24.660965 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:24.661483 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:24.661489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:24.661492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:24.661956 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:24.662280 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.663023 (+ 743us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.663303 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:01:24.664727 23438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:24.665160 23090 catalog_manager.cc:4841] TS c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723): tablet f02aa07e99f948d1b0ef03f52b070512 (table test-table [id=6d9aeb0a4dbc498aa3cc4c565ea62dac]) successfully deleted
I20260808 04:01:24.790776 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 23219
I20260808 04:01:24.826421 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 23458
Aug 08 04:01:24 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 04:01:24.845084 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.827311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.28.251.194:42723)
0808 04:01:24.827561 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.827565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.827587 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:01:24.827773 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.827778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.827853 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.827937 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:24.829978 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.830897 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.833033 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.833538 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.833671 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:24.838353 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:24.838376 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:24.838397 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:24.838465 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:24.841901 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:24.842705 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:24.842716 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:24.842718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:24.842884 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:24.843308 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:24.843312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:24.843314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:24.843810 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:24.844150 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:24.844496 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 04:01:24.844753 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 04:01:24.845885 23659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:24.964994 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 23458
I20260808 04:01:24.992729 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 23674
Aug 08 04:01:25 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 04:01:25.013880 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:24.993497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.28.251.195:36103)
0808 04:01:24.993687 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:24.993693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:24.993716 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:01:24.993942 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:24.993946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:24.994024 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:24.994108 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:24.995441 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.996241 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.997673 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:24.998080 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:24.998197 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:25.007313 (+ 9116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:25.007336 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:25.007355 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:25.007405 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:25.010752 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:25.011639 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:25.011648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:25.011651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:25.011812 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:25.012233 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:25.012237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:25.012239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:25.012625 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:25.012932 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:25.013274 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 04:01:25.013510 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:01:25.014607 23872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:25.107808 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 23674
I20260808 04:01:25.137801 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 23063
I20260808 04:01:25.139334 23166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:25.254653 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 23063
I20260808 04:01:45.283833 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41725
--webserver_interface=127.28.251.254
--webserver_port=46533
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:41725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:01:45.652935 23965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:45.653250 23965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:45.653339 23965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:45.660715 23965 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:01:45.660832 23965 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:01:45.660880 23965 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:01:45.660933 23965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:01:45.660988 23965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:45.661056 23965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:45.661103 23965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:01:45.661156 23965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:01:45.672564 23965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:41725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=46533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:45.673959 23965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:45.675347 23965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:45.701980 23977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:45.701957 23983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:45.701923 23978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:45.702833 23965 server_base.cc:1034] running on GCE node
I20260808 04:01:45.703894 23965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:45.705684 23965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:45.706957 23965 hybrid_clock.cc:648] HybridClock initialized: now 1786161705706874 us; error 56 us; skew 500 ppm
I20260808 04:01:45.707427 23965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:45 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161705, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:45.711352 23965 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:01:45.713195 23965 webserver.cc:460] Webserver started at http://127.28.251.254:46533/ using document root <none> and password file <none>
I20260808 04:01:45.714046 23965 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:45.714175 23965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:45.720858 23965 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:01:45.724215 23992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:45.725752 23965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:01:45.725945 23965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "7f28a8afb50749e5ae40c4d6e953da93"
format_stamp: "Formatted at 2026-08-08 04:01:22 on dist-test-slave-pww0"
server_key: "26305652e8ee525f850c439151b8e7f7"
server_key_iv: "22e3eaf95821fab825aac237fbb005dc"
server_key_version: "encryptionkey@0"
I20260808 04:01:45.727058 23965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:45.752233 23965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:45.756918 23965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:45.757229 23965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:45.777002 23965 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:41725
I20260808 04:01:45.777045 24080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:41725 every 8 connection(s)
I20260808 04:01:45.778599 23965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:01:45.781277 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 23965
I20260808 04:01:45.782128 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:44809
--local_ip_for_outbound_sockets=127.28.251.193
--tserver_master_addrs=127.28.251.254:41725
--webserver_port=34419
--webserver_interface=127.28.251.193
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:01:45.786926 24083 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:01:45.791080 24083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Bootstrap starting.
I20260808 04:01:45.813961 24083 log.cc:824] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:45.828794 24083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:01:45.829458 24083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Bootstrap complete.
I20260808 04:01:45.839934 24083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:01:45.840956 24083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f28a8afb50749e5ae40c4d6e953da93, State: Initialized, Role: FOLLOWER
I20260808 04:01:45.841753 24083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:01:45.842031 24083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:01:45.842180 24083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:01:45.842366 24083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:01:45.846935 24083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:01:45.847645 24083 leader_election.cc:302] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f28a8afb50749e5ae40c4d6e953da93; no voters:
I20260808 04:01:45.848348 24083 leader_election.cc:288] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:01:45.848570 24087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:01:45.849856 24087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 2 LEADER]: Becoming Leader. State: Replica: 7f28a8afb50749e5ae40c4d6e953da93, State: Running, Role: LEADER
I20260808 04:01:45.850602 24087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:01:45.851123 24083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:01:45.853372 24090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f28a8afb50749e5ae40c4d6e953da93. Latest consensus state: current_term: 2 leader_uuid: "7f28a8afb50749e5ae40c4d6e953da93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } } }
I20260808 04:01:45.853806 24090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: This master's current role is: LEADER
I20260808 04:01:45.854457 24088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7f28a8afb50749e5ae40c4d6e953da93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } } }
I20260808 04:01:45.854795 24088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: This master's current role is: LEADER
I20260808 04:01:45.860105 24095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:01:45.870164 24095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:01:45.871907 24095 catalog_manager.cc:1259] Loaded cluster ID: 022c774db10a4670bcef7985358315ba
I20260808 04:01:45.872035 24095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:01:45.876389 24095 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:01:45.878155 24095 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Loaded TSK: 0
I20260808 04:01:45.910245 24095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Generated new TSK 1
I20260808 04:01:45.910763 24095 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:01:46.166405 24085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:46.166775 24085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:46.166939 24085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:46.174678 24085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:46.174847 24085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:46.175019 24085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:01:46.186175 24085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:44809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=34419
--tserver_master_addrs=127.28.251.254:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:46.187686 24085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:46.189114 24085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:46.206539 24124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:46.206565 24125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:46.207928 24131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:46.207921 24085 server_base.cc:1034] running on GCE node
I20260808 04:01:46.208832 24085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:46.210141 24085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:46.211391 24085 hybrid_clock.cc:648] HybridClock initialized: now 1786161706211327 us; error 74 us; skew 500 ppm
I20260808 04:01:46.211800 24085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:46 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161706, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:46.215857 24085 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 04:01:46.217903 24085 webserver.cc:460] Webserver started at http://127.28.251.193:34419/ using document root <none> and password file <none>
I20260808 04:01:46.218902 24085 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:46.219066 24085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:46.226774 24085 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:01:46.230569 24147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:46.232538 24085 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:01:46.232755 24085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "e161f3da09a342d7afab36246bccebbd"
format_stamp: "Formatted at 2026-08-08 04:01:23 on dist-test-slave-pww0"
server_key: "b377483e6428c897ba3a839dc20c4546"
server_key_iv: "3439b9ad267f2b4b2a859f2f336ba0b1"
server_key_version: "encryptionkey@0"
I20260808 04:01:46.234196 24085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:46.260207 24085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:46.266150 24085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:46.266512 24085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:46.267971 24085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:46.269529 24085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:46.269658 24085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:46.269757 24085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:46.269826 24085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:46.305295 24085 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:44809
I20260808 04:01:46.305383 24341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:44809 every 8 connection(s)
I20260808 04:01:46.306982 24085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:01:46.308718 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 24085
I20260808 04:01:46.309489 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:42723
--local_ip_for_outbound_sockets=127.28.251.194
--tserver_master_addrs=127.28.251.254:41725
--webserver_port=43591
--webserver_interface=127.28.251.194
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:01:46 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161706, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:01:46.334759 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:46.310258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:37019 (local address 127.28.251.254:41725)
0808 04:01:46.312472 (+ 2214us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:46.312484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:46.312542 (+ 58us) server_negotiation.cc:407] Connection header received
0808 04:01:46.314683 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:46.314700 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:46.315140 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:46.315544 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:46.317862 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:46.319171 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:46.321544 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:46.322282 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:46.322453 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:46.326592 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:46.326601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:46.326621 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:46.326676 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:46.329572 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:46.330607 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:46.330615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:46.330621 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:46.330804 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:46.331443 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:46.331447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:46.331450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:46.332095 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:46.332260 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:46.333333 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0808 04:01:46.333718 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1998,"thread_start_us":1850,"threads_started":1}
I20260808 04:01:46.335137 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:46.310438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:37019)
0808 04:01:46.311519 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:46.311577 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:46.314025 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:46.315822 (+ 1797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:46.315834 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:46.316672 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:01:46.317553 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:46.317584 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:46.319398 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:46.319404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:46.321324 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:46.321344 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:46.322567 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:46.322575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:46.322877 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:46.323746 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:01:46.323771 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:01:46.326234 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:01:46.329756 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:46.329768 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:46.329785 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:46.330457 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:46.330928 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:46.330933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:46.330936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:46.331252 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:46.332253 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:01:46.332262 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:01:46.332721 (+ 459us) client_negotiation.cc:769] Sending connection context
0808 04:01:46.333127 (+ 406us) client_negotiation.cc:240] Negotiation successful
0808 04:01:46.333758 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":393,"threads_started":1}
I20260808 04:01:46.338150 24342 heartbeater.cc:347] Connected to a master server at 127.28.251.254:41725
I20260808 04:01:46.338490 24342 heartbeater.cc:464] Registering TS with master...
I20260808 04:01:46.339390 24342 heartbeater.cc:511] Master 127.28.251.254:41725 requested a full tablet report, sending...
I20260808 04:01:46.341769 24020 ts_manager.cc:194] Registered new tserver with Master: e161f3da09a342d7afab36246bccebbd (127.28.251.193:44809)
I20260808 04:01:46.343523 24020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:37019
W20260808 04:01:46.678095 24346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:46.678407 24346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:46.678542 24346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:46.686262 24346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:46.686398 24346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:46.686545 24346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:01:46.696871 24346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:42723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=43591
--tserver_master_addrs=127.28.251.254:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:46.698519 24346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:46.699991 24346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:46.714342 24356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:46.714448 24355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:46.714310 24362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:46.714877 24346 server_base.cc:1034] running on GCE node
I20260808 04:01:46.715801 24346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:46.716856 24346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:46.718061 24346 hybrid_clock.cc:648] HybridClock initialized: now 1786161706717990 us; error 61 us; skew 500 ppm
I20260808 04:01:46.718408 24346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:46 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161706, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:46.721871 24346 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 04:01:46.723615 24346 webserver.cc:460] Webserver started at http://127.28.251.194:43591/ using document root <none> and password file <none>
I20260808 04:01:46.724355 24346 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:46.724495 24346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:46.730806 24346 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 04:01:46.733970 24371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:46.735620 24346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:01:46.735805 24346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "c2956e6c23b842d8be00d1e2c200b612"
format_stamp: "Formatted at 2026-08-08 04:01:23 on dist-test-slave-pww0"
server_key: "97e112a3e729ea4a358ebadcb5f5f125"
server_key_iv: "2cff378ded54b402c67156dccd93677e"
server_key_version: "encryptionkey@0"
I20260808 04:01:46.736891 24346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:46.760711 24346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:46.765528 24346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:46.765836 24346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:46.766987 24346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:46.768592 24346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:46.768693 24346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:46.768823 24346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:46.768896 24346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:46.802239 24346 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:42723
I20260808 04:01:46.802315 24551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:42723 every 8 connection(s)
I20260808 04:01:46.803860 24346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:01:46.809430 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 24346
I20260808 04:01:46.810046 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:36103
--local_ip_for_outbound_sockets=127.28.251.195
--tserver_master_addrs=127.28.251.254:41725
--webserver_port=45759
--webserver_interface=127.28.251.195
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:01:46 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161706, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:01:46.827360 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:46.806800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:52811 (local address 127.28.251.254:41725)
0808 04:01:46.806989 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:46.806993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:46.808245 (+ 1252us) server_negotiation.cc:407] Connection header received
0808 04:01:46.810284 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:46.810290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:46.810433 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:46.810541 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:46.812212 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:46.813155 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:46.815256 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:46.815879 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:46.816013 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:46.820585 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:46.820592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:46.820597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:46.820640 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:46.823513 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:46.824541 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:46.824547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:46.824550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:46.824760 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:46.825353 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:46.825358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:46.825360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:46.825696 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:46.825859 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:46.826719 (+ 860us) server_negotiation.cc:299] Negotiation successful
0808 04:01:46.826999 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:01:46.828459 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:46.807276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:52811)
0808 04:01:46.808054 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:46.808103 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:46.809966 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:46.810780 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:46.810788 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:46.811356 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:01:46.812018 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:46.812039 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:46.813374 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:46.813379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:46.815080 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:46.815095 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:46.816116 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:46.816122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:46.816388 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:46.817156 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:01:46.817191 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:01:46.820268 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:01:46.823712 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:46.823723 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:46.823743 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:46.824344 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:46.824892 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:46.824897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:46.824900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:46.825196 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:46.825862 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:01:46.825873 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:01:46.826309 (+ 436us) client_negotiation.cc:769] Sending connection context
0808 04:01:46.826583 (+ 274us) client_negotiation.cc:240] Negotiation successful
0808 04:01:46.827195 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":322,"threads_started":1}
I20260808 04:01:46.830065 24552 heartbeater.cc:347] Connected to a master server at 127.28.251.254:41725
I20260808 04:01:46.830415 24552 heartbeater.cc:464] Registering TS with master...
I20260808 04:01:46.831244 24552 heartbeater.cc:511] Master 127.28.251.254:41725 requested a full tablet report, sending...
I20260808 04:01:46.832893 24020 ts_manager.cc:194] Registered new tserver with Master: c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723)
I20260808 04:01:46.833702 24020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:52811
W20260808 04:01:47.189849 24556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:01:47.190183 24556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:01:47.190335 24556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:01:47.198052 24556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:01:47.198202 24556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:01:47.198370 24556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:01:47.209313 24556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:36103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=45759
--tserver_master_addrs=127.28.251.254:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:01:47.210747 24556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:01:47.212214 24556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:01:47.226296 24566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:47.226404 24569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:01:47.226418 24565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:01:47.227351 24556 server_base.cc:1034] running on GCE node
I20260808 04:01:47.228315 24556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:01:47.229369 24556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:01:47.230602 24556 hybrid_clock.cc:648] HybridClock initialized: now 1786161707230540 us; error 49 us; skew 500 ppm
I20260808 04:01:47.231003 24556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:01:47 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161707, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:01:47.234525 24556 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 04:01:47.236191 24556 webserver.cc:460] Webserver started at http://127.28.251.195:45759/ using document root <none> and password file <none>
I20260808 04:01:47.236945 24556 fs_manager.cc:359] Metadata directory not provided
I20260808 04:01:47.237074 24556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:01:47.243163 24556 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:01:47.246906 24583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:47.248474 24556 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:01:47.248659 24556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "03148b5891fe444dac3625203b23adb9"
format_stamp: "Formatted at 2026-08-08 04:01:24 on dist-test-slave-pww0"
server_key: "50eba8b20b7ca7e8c4233dbe2b2123c7"
server_key_iv: "b179bd56d6ee544c7ae2c5e83563fa65"
server_key_version: "encryptionkey@0"
I20260808 04:01:47.249735 24556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:01:47.281368 24556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:01:47.286033 24556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:01:47.286322 24556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:01:47.287446 24556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:01:47.288950 24556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:01:47.289064 24556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:47.289177 24556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:01:47.289250 24556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:01:47.321846 24556 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:36103
I20260808 04:01:47.321926 24773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:36103 every 8 connection(s)
I20260808 04:01:47.323366 24556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:01:47.329741 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 24556
Aug 08 04:01:47 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161707, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:01:47.346951 24342 heartbeater.cc:503] Master 127.28.251.254:41725 was elected leader, sending a full tablet report...
I20260808 04:01:47.347262 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.330843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.28.251.254:41725)
0808 04:01:47.331683 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.331687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.331705 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:01:47.331792 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.331798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.331857 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.331952 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:47.333891 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.334654 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.336389 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.336910 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.337008 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.340506 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:47.340514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:47.340520 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:47.340591 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:47.343845 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:47.344866 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:47.344874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:47.344878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:47.345050 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:47.345529 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:47.345533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:47.345535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:47.345916 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:47.346194 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:47.346605 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 04:01:47.346846 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"spinlock_wait_cycles":1280,"thread_start_us":633,"threads_started":1}
I20260808 04:01:47.349452 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.326143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:37913 (local address 127.28.251.254:41725)
0808 04:01:47.326360 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.326364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.327864 (+ 1500us) server_negotiation.cc:407] Connection header received
0808 04:01:47.330039 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.330045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.330146 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.330242 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:47.332731 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.333507 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.335727 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.336201 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.336311 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.341057 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:47.341062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:47.341066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:47.341117 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:47.344399 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:47.345459 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:47.345465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:47.345468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:47.345618 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:47.346227 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:47.346233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:47.346236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:47.346598 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:47.346735 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:47.348863 (+ 2128us) server_negotiation.cc:299] Negotiation successful
0808 04:01:47.349080 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 04:01:47.350258 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.326720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:37913)
0808 04:01:47.327670 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:47.327723 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:47.329684 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:47.330584 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:47.330598 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:47.331509 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:01:47.332505 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.332538 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.333717 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.333722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.335532 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.335551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.336463 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.336469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.336699 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.337487 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:01:47.337521 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:01:47.340722 (+ 3201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:01:47.344626 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:47.344639 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:47.344662 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:47.345321 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:47.345768 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:01:47.345772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:01:47.345776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:01:47.346073 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:01:47.347351 (+ 1278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:01:47.347360 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:01:47.347937 (+ 577us) client_negotiation.cc:769] Sending connection context
0808 04:01:47.348259 (+ 322us) client_negotiation.cc:240] Negotiation successful
0808 04:01:47.348889 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":396,"threads_started":1}
I20260808 04:01:47.351714 24774 heartbeater.cc:347] Connected to a master server at 127.28.251.254:41725
I20260808 04:01:47.352070 24774 heartbeater.cc:464] Registering TS with master...
I20260808 04:01:47.352857 24774 heartbeater.cc:511] Master 127.28.251.254:41725 requested a full tablet report, sending...
I20260808 04:01:47.354374 24020 ts_manager.cc:194] Registered new tserver with Master: 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103)
I20260808 04:01:47.355249 24020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:37913
I20260808 04:01:47.362946 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:01:47.369596 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.363995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.28.251.254:41725)
0808 04:01:47.364217 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.364224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.364642 (+ 418us) server_negotiation.cc:407] Connection header received
0808 04:01:47.364832 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.364836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.364987 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.365117 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:01:47.365490 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.366249 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.367790 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.368225 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.368329 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.368830 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:01:47.369066 (+ 236us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 04:01:47.369376 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55882: token expired
Metrics: {"server-negotiator.queue_time_us":83}
W20260808 04:01:47.369745 24344 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55882: token expired
W20260808 04:01:47.370415 23890 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.251.254:41725)
I20260808 04:01:47.384447 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.370920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.28.251.254:41725)
0808 04:01:47.371068 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.371072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.371089 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:01:47.371203 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.371207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.371281 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.371365 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:47.372726 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.373483 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.375112 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.375613 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.375738 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.378826 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:47.378832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:47.378835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:47.378878 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:47.381636 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:47.382520 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:47.382524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:47.382526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:47.382658 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:47.383006 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:47.383009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:47.383011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:47.383375 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:47.383652 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:47.383979 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 04:01:47.384180 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:01:47.388216 24020 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:01:47.395952 24020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:01:47.419551 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.408989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.28.251.194:42723)
0808 04:01:47.409674 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.409682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.410699 (+ 1017us) server_negotiation.cc:407] Connection header received
0808 04:01:47.411044 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.411053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.411311 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.411511 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:47.412668 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.414365 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.416729 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.418246 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.418434 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.418627 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:47.418960 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 04:01:47.419147 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":360,"threads_started":1}
I20260808 04:01:47.420202 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.409452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:42723 (local address 127.0.0.1:60866)
0808 04:01:47.410581 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:47.410595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:47.410773 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:47.411814 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:47.411823 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:47.411864 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:47.412428 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.412446 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.414636 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.414642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.416556 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.416569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.418474 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.418480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.418682 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.418737 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 04:01:47.418894 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 04:01:47.419107 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":97,"spinlock_wait_cycles":14080,"thread_start_us":844,"threads_started":1}
I20260808 04:01:47.420308 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.410762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:36103 (local address 127.0.0.1:36428)
0808 04:01:47.411453 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:47.411511 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:47.411630 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:47.412769 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:47.412774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:47.412800 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:47.413207 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.413220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.415596 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.415601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.417559 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.417572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.419486 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.419493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.419669 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.419702 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:01:47.419789 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 04:01:47.419928 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":757,"threads_started":1}
I20260808 04:01:47.420709 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.408669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.28.251.193:44809)
0808 04:01:47.409368 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.409378 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.410776 (+ 1398us) server_negotiation.cc:407] Connection header received
0808 04:01:47.411044 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.411053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.411309 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.411515 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:47.412686 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.414344 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.416928 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.418952 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.419151 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.419319 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:47.420206 (+ 887us) server_negotiation.cc:299] Negotiation successful
0808 04:01:47.420395 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":317,"threads_started":1}
I20260808 04:01:47.420895 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.408472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.0.0.1:48702)
0808 04:01:47.410517 (+ 2045us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:47.410575 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:47.410773 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:47.411814 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:47.411823 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:47.411857 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:47.412505 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.412515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.414513 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.414519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.416757 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.416772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.419532 (+ 2760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.419540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.419732 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.419764 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:01:47.419945 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 04:01:47.420085 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1504,"mutex_wait_us":85,"spinlock_wait_cycles":12032,"thread_start_us":1359,"threads_started":1}
I20260808 04:01:47.421609 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.411250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.28.251.195:36103)
0808 04:01:47.412177 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.412183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.412210 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:01:47.412286 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.412299 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.412521 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.413466 (+ 945us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:47.413761 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.415392 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.417705 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.419167 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.419439 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.419633 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:47.421094 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0808 04:01:47.421271 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":635,"threads_started":1}
I20260808 04:01:47.424587 24451 tablet_service.cc:1467] Processing CreateTablet for tablet a8bed27dbaff4a2794bd0263376cbdc7 (DEFAULT_TABLE table=test-table [id=efdbe42763314190a66aa7d524fe414a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:01:47.424742 24224 tablet_service.cc:1467] Processing CreateTablet for tablet a8bed27dbaff4a2794bd0263376cbdc7 (DEFAULT_TABLE table=test-table [id=efdbe42763314190a66aa7d524fe414a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:01:47.425266 24662 tablet_service.cc:1467] Processing CreateTablet for tablet a8bed27dbaff4a2794bd0263376cbdc7 (DEFAULT_TABLE table=test-table [id=efdbe42763314190a66aa7d524fe414a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:01:47.425873 24451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8bed27dbaff4a2794bd0263376cbdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:01:47.426206 24224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8bed27dbaff4a2794bd0263376cbdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:01:47.426657 24662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8bed27dbaff4a2794bd0263376cbdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:01:47.439121 24799 tablet_bootstrap.cc:492] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: Bootstrap starting.
I20260808 04:01:47.443761 24799 tablet_bootstrap.cc:654] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: Neither blocks nor log segments found. Creating new log.
I20260808 04:01:47.444221 24802 tablet_bootstrap.cc:492] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: Bootstrap starting.
I20260808 04:01:47.445371 24799 log.cc:824] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:47.446260 24800 tablet_bootstrap.cc:492] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Bootstrap starting.
I20260808 04:01:47.448076 24802 tablet_bootstrap.cc:654] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: Neither blocks nor log segments found. Creating new log.
I20260808 04:01:47.448597 24799 tablet_bootstrap.cc:492] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: No bootstrap required, opened a new log
I20260808 04:01:47.448884 24799 ts_tablet_manager.cc:1397] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 04:01:47.449429 24802 log.cc:824] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:47.450053 24800 tablet_bootstrap.cc:654] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Neither blocks nor log segments found. Creating new log.
I20260808 04:01:47.451436 24800 log.cc:824] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:01:47.453056 24802 tablet_bootstrap.cc:492] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: No bootstrap required, opened a new log
I20260808 04:01:47.453359 24802 ts_tablet_manager.cc:1397] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 04:01:47.455046 24800 tablet_bootstrap.cc:492] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: No bootstrap required, opened a new log
I20260808 04:01:47.455354 24800 ts_tablet_manager.cc:1397] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260808 04:01:47.457535 24799 raft_consensus.cc:348] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:01:47.457948 24799 raft_consensus.cc:374] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:01:47.458093 24799 raft_consensus.cc:729] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2956e6c23b842d8be00d1e2c200b612, State: Initialized, Role: FOLLOWER
I20260808 04:01:47.458726 24799 consensus_queue.cc:260] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:01:47.459988 24552 heartbeater.cc:503] Master 127.28.251.254:41725 was elected leader, sending a full tablet report...
I20260808 04:01:47.460871 24799 ts_tablet_manager.cc:1428] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260808 04:01:47.465775 24802 raft_consensus.cc:348] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:01:47.466337 24802 raft_consensus.cc:374] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:01:47.466053 24800 raft_consensus.cc:348] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:01:47.466450 24802 raft_consensus.cc:729] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03148b5891fe444dac3625203b23adb9, State: Initialized, Role: FOLLOWER
I20260808 04:01:47.466488 24800 raft_consensus.cc:374] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:01:47.466584 24800 raft_consensus.cc:729] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e161f3da09a342d7afab36246bccebbd, State: Initialized, Role: FOLLOWER
I20260808 04:01:47.467175 24802 consensus_queue.cc:260] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:01:47.467175 24800 consensus_queue.cc:260] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:01:47.468900 24774 heartbeater.cc:503] Master 127.28.251.254:41725 was elected leader, sending a full tablet report...
I20260808 04:01:47.469203 24800 ts_tablet_manager.cc:1428] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260808 04:01:47.469493 24802 ts_tablet_manager.cc:1428] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260808 04:01:47.556191 24553 tablet.cc:2369] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:01:47.559880 24343 tablet.cc:2369] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:01:47.575071 24775 tablet.cc:2369] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:01:47.626276 24813 raft_consensus.cc:482] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:01:47.626646 24813 raft_consensus.cc:504] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:01:47.628602 24813 leader_election.cc:288] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723), 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103)
I20260808 04:01:47.635700 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.629139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:37749 (local address 127.28.251.194:42723)
0808 04:01:47.629370 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.629375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.629967 (+ 592us) server_negotiation.cc:407] Connection header received
0808 04:01:47.630261 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.630268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.630368 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.630483 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:47.631175 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.632170 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.633966 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.634766 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.634913 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.634995 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:47.635283 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 04:01:47.635414 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 04:01:47.635849 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.629119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:42723 (local address 127.28.251.193:37749)
0808 04:01:47.629834 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:47.629854 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:47.630008 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:47.630575 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:47.630580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:47.630607 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:47.631011 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.631024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.632300 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.632306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.633808 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.633821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.634978 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.634984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.635127 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.635154 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:01:47.635237 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 04:01:47.635377 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":25,"thread_start_us":380,"threads_started":1}
I20260808 04:01:47.636209 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.630290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:54785 (local address 127.28.251.195:36103)
0808 04:01:47.630515 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.630520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.630863 (+ 343us) server_negotiation.cc:407] Connection header received
0808 04:01:47.631012 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.631016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.631090 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.631184 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:01:47.631792 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.632593 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.634397 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.635240 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.635391 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.635491 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:47.635831 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 04:01:47.635963 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 04:01:47.636330 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.630026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:36103 (local address 127.28.251.193:54785)
0808 04:01:47.630745 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:47.630765 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:01:47.630891 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:01:47.631205 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:01:47.631209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:01:47.631235 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:01:47.631628 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.631651 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.632773 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.632780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.634241 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:01:47.634255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.635508 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.635516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:01:47.635673 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.635698 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:01:47.635784 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 04:01:47.635925 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":4,"thread_start_us":322,"threads_started":1}
I20260808 04:01:47.637001 24485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8bed27dbaff4a2794bd0263376cbdc7" candidate_uuid: "e161f3da09a342d7afab36246bccebbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2956e6c23b842d8be00d1e2c200b612" is_pre_election: true
I20260808 04:01:47.637559 24485 raft_consensus.cc:2454] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e161f3da09a342d7afab36246bccebbd in term 0.
I20260808 04:01:47.637454 24701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8bed27dbaff4a2794bd0263376cbdc7" candidate_uuid: "e161f3da09a342d7afab36246bccebbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03148b5891fe444dac3625203b23adb9" is_pre_election: true
I20260808 04:01:47.637943 24701 raft_consensus.cc:2454] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e161f3da09a342d7afab36246bccebbd in term 0.
I20260808 04:01:47.638327 24150 leader_election.cc:302] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2956e6c23b842d8be00d1e2c200b612, e161f3da09a342d7afab36246bccebbd; no voters:
I20260808 04:01:47.638821 24813 raft_consensus.cc:2781] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:01:47.638989 24813 raft_consensus.cc:482] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:01:47.639099 24813 raft_consensus.cc:3037] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:01:47.641665 24813 raft_consensus.cc:504] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:01:47.642293 24813 leader_election.cc:288] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 1 election: Requested vote from peers c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723), 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103)
I20260808 04:01:47.643076 24485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8bed27dbaff4a2794bd0263376cbdc7" candidate_uuid: "e161f3da09a342d7afab36246bccebbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2956e6c23b842d8be00d1e2c200b612"
I20260808 04:01:47.643076 24701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8bed27dbaff4a2794bd0263376cbdc7" candidate_uuid: "e161f3da09a342d7afab36246bccebbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03148b5891fe444dac3625203b23adb9"
I20260808 04:01:47.643327 24485 raft_consensus.cc:3037] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:01:47.643326 24701 raft_consensus.cc:3037] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:01:47.646756 24701 raft_consensus.cc:2454] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e161f3da09a342d7afab36246bccebbd in term 1.
I20260808 04:01:47.646756 24485 raft_consensus.cc:2454] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e161f3da09a342d7afab36246bccebbd in term 1.
I20260808 04:01:47.647354 24150 leader_election.cc:302] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03148b5891fe444dac3625203b23adb9, e161f3da09a342d7afab36246bccebbd; no voters:
I20260808 04:01:47.647800 24813 raft_consensus.cc:2781] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:01:47.648312 24813 raft_consensus.cc:686] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 LEADER]: Becoming Leader. State: Replica: e161f3da09a342d7afab36246bccebbd, State: Running, Role: LEADER
I20260808 04:01:47.649006 24813 consensus_queue.cc:237] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:01:47.654248 24020 catalog_manager.cc:5495] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd reported cstate change: term changed from 0 to 1, leader changed from <none> to e161f3da09a342d7afab36246bccebbd (127.28.251.193). New cstate: current_term: 1 leader_uuid: "e161f3da09a342d7afab36246bccebbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20260808 04:01:47.720801 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.714788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.28.251.193:44809)
0808 04:01:47.715046 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.715050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.715149 (+ 99us) server_negotiation.cc:407] Connection header received
0808 04:01:47.715409 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.715414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.715565 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.715718 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:01:47.716104 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.716859 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.718333 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.718803 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.718932 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.719567 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:01:47.720167 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:01:47.720288 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:47.720437 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 04:01:47.720523 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260808 04:01:47.730468 24701 raft_consensus.cc:1261] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9 [term 1 FOLLOWER]: Refusing update from remote peer e161f3da09a342d7afab36246bccebbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:01:47.730468 24485 raft_consensus.cc:1261] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612 [term 1 FOLLOWER]: Refusing update from remote peer e161f3da09a342d7afab36246bccebbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:01:47.731751 24813 consensus_queue.cc:1035] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:01:47.732126 24817 consensus_queue.cc:1035] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:01:47.745121 24821 mvcc.cc:205] Tried to move back new op lower bound from 7316118354850222080 to 7316118354534821888. Current Snapshot: MvccSnapshot[applied={T|T < 7316118354850222080}]
I20260808 04:01:47.748673 24822 mvcc.cc:205] Tried to move back new op lower bound from 7316118354850222080 to 7316118354534821888. Current Snapshot: MvccSnapshot[applied={T|T < 7316118354850222080}]
I20260808 04:01:47.762146 24020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:01:47.762496 24020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:01:47.766603 24020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Sending DeleteTablet for 3 replicas of tablet a8bed27dbaff4a2794bd0263376cbdc7
I20260808 04:01:47.767951 24224 tablet_service.cc:1514] Processing DeleteTablet for tablet a8bed27dbaff4a2794bd0263376cbdc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:01:47 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:48702
I20260808 04:01:47.768070 24451 tablet_service.cc:1514] Processing DeleteTablet for tablet a8bed27dbaff4a2794bd0263376cbdc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:01:47 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:60866
I20260808 04:01:47.768366 24662 tablet_service.cc:1514] Processing DeleteTablet for tablet a8bed27dbaff4a2794bd0263376cbdc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:01:47 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:36428
I20260808 04:01:47.769244 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 24085
I20260808 04:01:47.795104 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.770259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.28.251.193:44809)
0808 04:01:47.770465 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.770471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.770493 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:01:47.770678 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.770683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.770771 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.770909 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:47.773157 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.774407 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.776522 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.777225 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.777379 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.783658 (+ 6279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:47.783679 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:47.783704 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:47.783770 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:47.788228 (+ 4458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:47.789508 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:47.789521 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:47.789529 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:47.789749 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:47.791380 (+ 1631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:47.791387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:47.791390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:47.792007 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:47.792356 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:47.794356 (+ 2000us) server_negotiation.cc:299] Negotiation successful
0808 04:01:47.794651 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:01:47.797591 24336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:47.799286 24845 tablet_replica.cc:330] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: stopping tablet replica
I20260808 04:01:47.799918 24845 raft_consensus.cc:2229] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:01:47.800705 24845 raft_consensus.cc:2258] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:01:47.801802 24841 tablet_replica.cc:330] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: stopping tablet replica
I20260808 04:01:47.802366 24841 raft_consensus.cc:2229] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:01:47.802974 24841 raft_consensus.cc:2258] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:01:47.805704 24841 ts_tablet_manager.cc:1905] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:01:47.814155 24841 ts_tablet_manager.cc:1918] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:01:47.814447 24841 log.cc:1196] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/a8bed27dbaff4a2794bd0263376cbdc7
I20260808 04:01:47.815100 24841 ts_tablet_manager.cc:1939] T a8bed27dbaff4a2794bd0263376cbdc7 P c2956e6c23b842d8be00d1e2c200b612: Deleting consensus metadata
I20260808 04:01:47.815284 24845 ts_tablet_manager.cc:1905] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:01:47.817284 23994 catalog_manager.cc:4841] TS c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723): tablet a8bed27dbaff4a2794bd0263376cbdc7 (table test-table [id=efdbe42763314190a66aa7d524fe414a]) successfully deleted
I20260808 04:01:47.823854 24845 ts_tablet_manager.cc:1918] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:01:47.824137 24845 log.cc:1196] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/a8bed27dbaff4a2794bd0263376cbdc7
I20260808 04:01:47.824692 24845 ts_tablet_manager.cc:1939] T a8bed27dbaff4a2794bd0263376cbdc7 P 03148b5891fe444dac3625203b23adb9: Deleting consensus metadata
I20260808 04:01:47.826854 23994 catalog_manager.cc:4841] TS 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103): tablet a8bed27dbaff4a2794bd0263376cbdc7 (table test-table [id=efdbe42763314190a66aa7d524fe414a]) successfully deleted
I20260808 04:01:47.921478 24840 tablet_replica.cc:330] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: stopping tablet replica
I20260808 04:01:47.922076 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 24085
I20260808 04:01:47.922086 24840 raft_consensus.cc:2229] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 LEADER]: Raft consensus shutting down.
W20260808 04:01:47.945531 23993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.251.193:44809 (error 108)
I20260808 04:01:47.945760 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 24346
I20260808 04:01:47.947885 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.947110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.0.0.1:48726)
0808 04:01:47.947609 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:47.947716 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":234,"threads_started":1}
W20260808 04:01:47.948104 23993 catalog_manager.cc:4584] TS e161f3da09a342d7afab36246bccebbd (127.28.251.193:44809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8bed27dbaff4a2794bd0263376cbdc7: Network error: Client connection negotiation failed: client connection to 127.28.251.193:44809: connect: Connection refused (error 111)
I20260808 04:01:47.969018 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:47.946526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.28.251.194:42723)
0808 04:01:47.946705 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:47.946709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:47.946766 (+ 57us) server_negotiation.cc:407] Connection header received
0808 04:01:47.946941 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:47.946945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:47.947018 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:47.947102 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:47.948745 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.949508 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.950836 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:47.951310 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:47.951407 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:47.958436 (+ 7029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:47.958460 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:47.958477 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:47.958527 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:47.965783 (+ 7256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:47.966733 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:47.966744 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:47.966747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:47.966913 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:47.967314 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:47.967319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:47.967321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:47.967845 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:47.968116 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:47.968443 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 04:01:47.968651 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:01:47.969769 24547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:48.001997 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:48.001374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.0.0.1:48736)
0808 04:01:48.001648 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:48.001797 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 04:01:48.003288 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:48.002846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.0.0.1:48742)
0808 04:01:48.003010 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:48.003106 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 04:01:48.064173 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:48.063666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.0.0.1:48744)
0808 04:01:48.063880 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:48.063994 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 04:01:48.064205 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 24346
I20260808 04:01:48.070298 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:48.065285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.0.0.1:48750)
0808 04:01:48.070005 (+ 4720us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:48.070112 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4565}
I20260808 04:01:48.088806 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 24556
I20260808 04:01:48.103503 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:48.089636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.28.251.195:36103)
0808 04:01:48.089863 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 04:01:48.089868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:01:48.089888 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:01:48.090049 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:01:48.090054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:01:48.090136 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:01:48.090221 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:01:48.091686 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:48.092437 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:48.093931 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:01:48.094371 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:01:48.094486 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:01:48.097274 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:01:48.097310 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:01:48.097328 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:01:48.097372 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:01:48.100436 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:48.101260 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:48.101268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:48.101270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:48.101440 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:01:48.101839 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:01:48.101843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:01:48.101845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:01:48.102241 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:01:48.102527 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 04:01:48.102902 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 04:01:48.103133 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 04:01:48.104175 24767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:48.151260 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:48.150637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.0.0.1:48764)
0808 04:01:48.150916 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:48.151041 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260808 04:01:48.152585 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:48.152065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.0.0.1:48766)
0808 04:01:48.152292 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:48.152388 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260808 04:01:48.196153 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 24556
I20260808 04:01:48.224540 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 23965
I20260808 04:01:48.225826 24075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:01:48.341269 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:01:48.340283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:44809 (local address 127.0.0.1:48782)
0808 04:01:48.340910 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0808 04:01:48.341058 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
I20260808 04:01:48.341295 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 23965
I20260808 04:02:08.370644 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41725
--webserver_interface=127.28.251.254
--webserver_port=46533
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:41725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:08.744050 24856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:08.744356 24856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:08.744458 24856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:08.752223 24856 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:02:08.752341 24856 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:02:08.752382 24856 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:02:08.752456 24856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:02:08.752525 24856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:08.752624 24856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:08.752683 24856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:02:08.752758 24856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:02:08.763355 24856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:41725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=46533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:08.764827 24856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:08.766268 24856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:08.779065 24864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:08.779134 24867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:08.779104 24865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:08.779875 24856 server_base.cc:1034] running on GCE node
I20260808 04:02:08.780812 24856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:08.782411 24856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:08.783633 24856 hybrid_clock.cc:648] HybridClock initialized: now 1786161728783577 us; error 60 us; skew 500 ppm
I20260808 04:02:08.783975 24856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:08 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:08.793049 24856 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:02:08.794746 24856 webserver.cc:460] Webserver started at http://127.28.251.254:46533/ using document root <none> and password file <none>
I20260808 04:02:08.795583 24856 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:08.795701 24856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:08.801497 24856 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:02:08.804649 24885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:08.805995 24856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:02:08.806165 24856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "7f28a8afb50749e5ae40c4d6e953da93"
format_stamp: "Formatted at 2026-08-08 04:01:22 on dist-test-slave-pww0"
server_key: "26305652e8ee525f850c439151b8e7f7"
server_key_iv: "22e3eaf95821fab825aac237fbb005dc"
server_key_version: "encryptionkey@0"
I20260808 04:02:08.807148 24856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:08.826882 24856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:08.831571 24856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:08.831854 24856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:08.850871 24856 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:41725
I20260808 04:02:08.850898 24966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:41725 every 8 connection(s)
I20260808 04:02:08.852597 24856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:02:08.859825 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 24856
I20260808 04:02:08.860261 24967 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:02:08.860925 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:44809
--local_ip_for_outbound_sockets=127.28.251.193
--tserver_master_addrs=127.28.251.254:41725
--webserver_port=34419
--webserver_interface=127.28.251.193
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:02:08.865001 24967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Bootstrap starting.
I20260808 04:02:08.888943 24967 log.cc:824] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:08.905689 24967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:02:08.906279 24967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Bootstrap complete.
I20260808 04:02:08.915784 24967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:02:08.916705 24967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f28a8afb50749e5ae40c4d6e953da93, State: Initialized, Role: FOLLOWER
I20260808 04:02:08.917346 24967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:02:08.917580 24967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:02:08.917697 24967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:02:08.917865 24967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:02:08.921841 24967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:02:08.922343 24967 leader_election.cc:302] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f28a8afb50749e5ae40c4d6e953da93; no voters:
I20260808 04:02:08.923002 24967 leader_election.cc:288] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 04:02:08.923152 24971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:02:08.924365 24971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [term 3 LEADER]: Becoming Leader. State: Replica: 7f28a8afb50749e5ae40c4d6e953da93, State: Running, Role: LEADER
I20260808 04:02:08.925022 24971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } }
I20260808 04:02:08.925329 24967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:02:08.927660 24973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7f28a8afb50749e5ae40c4d6e953da93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } } }
I20260808 04:02:08.928072 24973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:08.928606 24976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f28a8afb50749e5ae40c4d6e953da93. Latest consensus state: current_term: 3 leader_uuid: "7f28a8afb50749e5ae40c4d6e953da93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f28a8afb50749e5ae40c4d6e953da93" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41725 } } }
I20260808 04:02:08.928910 24976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:08.940493 24987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:02:08.949713 24987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:02:08.951406 24987 catalog_manager.cc:1259] Loaded cluster ID: 022c774db10a4670bcef7985358315ba
I20260808 04:02:08.951634 24987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:02:08.955827 24987 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:02:08.957698 24987 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Loaded TSK: 0
I20260808 04:02:08.957979 24987 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f28a8afb50749e5ae40c4d6e953da93: Loaded TSK: 1
F20260808 04:02:08.959198 24987 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f76b03739ad google::LogMessage::Fail() at ??:0
@ 0x7f76b037688a google::LogMessage::SendToLog() at ??:0
@ 0x7f76b03734de google::LogMessage::Flush() at ??:0
@ 0x7f76b0374a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f76b40d77d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f76b40c673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f76c5da21c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f76c5da1783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f76c5e2d2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f76c5e2d0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f76c5e775e9 std::function<>::operator()() at ??:0
@ 0x7f76c5da5cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f76c5da3c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f76c5e2b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f76c5e2b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f76b2fbf345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f76b2fc291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f76b2fc2799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f76b2f9deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f76b62776db start_thread at ??:0
@ 0x7f76af20d61f clone at ??:0
W20260808 04:02:09.304534 24969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:09.304869 24969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:09.304996 24969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:09.313056 24969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:09.313199 24969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:09.313362 24969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:02:09.324402 24969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:44809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=34419
--tserver_master_addrs=127.28.251.254:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:09.325855 24969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:09.327277 24969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:09.341449 25048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:09.341478 25052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:09.341995 25047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:09.342051 24969 server_base.cc:1034] running on GCE node
I20260808 04:02:09.342825 24969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:09.343905 24969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:09.345139 24969 hybrid_clock.cc:648] HybridClock initialized: now 1786161729345085 us; error 42 us; skew 500 ppm
I20260808 04:02:09.345512 24969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:09 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:09.348999 24969 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 04:02:09.350651 24969 webserver.cc:460] Webserver started at http://127.28.251.193:34419/ using document root <none> and password file <none>
I20260808 04:02:09.351421 24969 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:09.351585 24969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:09.357568 24969 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:02:09.360823 25066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:09.362341 24969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:02:09.362524 24969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "e161f3da09a342d7afab36246bccebbd"
format_stamp: "Formatted at 2026-08-08 04:01:23 on dist-test-slave-pww0"
server_key: "b377483e6428c897ba3a839dc20c4546"
server_key_iv: "3439b9ad267f2b4b2a859f2f336ba0b1"
server_key_version: "encryptionkey@0"
I20260808 04:02:09.363610 24969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:09.401093 24969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:09.405547 24969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:09.405839 24969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:09.406917 24969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:09.408929 25075 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260808 04:02:09.415226 24969 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:02:09.415381 24969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260808 04:02:09.415542 24969 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:02:09.418938 24969 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:02:09.419034 24969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.000s
I20260808 04:02:09.419484 25075 tablet_bootstrap.cc:492] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Bootstrap starting.
I20260808 04:02:09.448387 25075 log.cc:824] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:09.460564 25075 tablet_bootstrap.cc:492] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:02:09.461267 25075 tablet_bootstrap.cc:492] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Bootstrap complete.
I20260808 04:02:09.462750 25075 ts_tablet_manager.cc:1397] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Time spent bootstrapping tablet: real 0.044s user 0.024s sys 0.015s
I20260808 04:02:09.466796 24969 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:44809
I20260808 04:02:09.466847 25246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:44809 every 8 connection(s)
I20260808 04:02:09.468755 24969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:02:09.470355 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 24969
I20260808 04:02:09.470968 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:42723
--local_ip_for_outbound_sockets=127.28.251.194
--tserver_master_addrs=127.28.251.254:41725
--webserver_port=43591
--webserver_interface=127.28.251.194
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:02:09.477130 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:09.474231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:54545)
0808 04:02:09.475321 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:09.475716 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":446,"threads_started":1}
W20260808 04:02:09.477939 25247 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:41725 (0 consecutive failures): Network error: Failed to ping master at 127.28.251.254:41725: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
I20260808 04:02:09.476894 25075 raft_consensus.cc:348] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:02:09.478454 25075 raft_consensus.cc:729] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e161f3da09a342d7afab36246bccebbd, State: Initialized, Role: FOLLOWER
I20260808 04:02:09.479074 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:09.478628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:60281)
0808 04:02:09.478823 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:09.478921 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 04:02:09.479152 25075 consensus_queue.cc:260] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:02:09.480355 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:09.479855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:42277)
0808 04:02:09.480086 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:09.480187 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260808 04:02:09.480558 25247 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:02:09.481021 25075 ts_tablet_manager.cc:1428] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
I20260808 04:02:09.483528 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:09.481005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:40333)
0808 04:02:09.483258 (+ 2253us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:09.483347 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2156,"threads_started":1}
I20260808 04:02:09.489909 25260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:09.489388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:43017)
0808 04:02:09.489579 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:09.489738 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 04:02:09.842883 25261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:09.843191 25261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:09.843290 25261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:09.850893 25261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:09.851022 25261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:09.851137 25261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:02:09.861490 25261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:42723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=43591
--tserver_master_addrs=127.28.251.254:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:09.862884 25261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:09.864275 25261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:09.878077 25274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:09.878151 25268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:09.878017 25269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:09.879293 25261 server_base.cc:1034] running on GCE node
I20260808 04:02:09.880051 25261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:09.881129 25261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:09.882367 25261 hybrid_clock.cc:648] HybridClock initialized: now 1786161729882301 us; error 49 us; skew 500 ppm
I20260808 04:02:09.882720 25261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:09 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:09.886214 25261 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 04:02:09.887998 25261 webserver.cc:460] Webserver started at http://127.28.251.194:43591/ using document root <none> and password file <none>
I20260808 04:02:09.888769 25261 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:09.888908 25261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:09.895578 25261 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20260808 04:02:09.898689 25290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:09.900270 25261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:02:09.900452 25261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "c2956e6c23b842d8be00d1e2c200b612"
format_stamp: "Formatted at 2026-08-08 04:01:23 on dist-test-slave-pww0"
server_key: "97e112a3e729ea4a358ebadcb5f5f125"
server_key_iv: "2cff378ded54b402c67156dccd93677e"
server_key_version: "encryptionkey@0"
I20260808 04:02:09.901520 25261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:09.926307 25261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:09.931576 25261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:09.931861 25261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:09.932967 25261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:09.934470 25261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:09.934566 25261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:09.934669 25261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:09.934744 25261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:09.968178 25261 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:42723
I20260808 04:02:09.968253 25491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:42723 every 8 connection(s)
I20260808 04:02:09.969736 25261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:02:09.971275 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 25261
I20260808 04:02:09.972046 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:36103
--local_ip_for_outbound_sockets=127.28.251.195
--tserver_master_addrs=127.28.251.254:41725
--webserver_port=45759
--webserver_interface=127.28.251.195
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:02:09.975524 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:09.973340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:49741)
0808 04:02:09.974239 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:09.974522 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":376,"threads_started":1}
W20260808 04:02:09.976325 25493 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:41725 (0 consecutive failures): Network error: Failed to ping master at 127.28.251.254:41725: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
I20260808 04:02:09.977372 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:09.976955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:59363)
0808 04:02:09.977151 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:09.977234 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 04:02:09.978309 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:09.977962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:35163)
0808 04:02:09.978109 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:09.978190 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 04:02:09.978557 25493 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:02:10.341028 25496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:10.341374 25496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:10.341521 25496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:10.349308 25496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:10.349457 25496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:10.349617 25496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:02:10.360005 25496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:36103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=45759
--tserver_master_addrs=127.28.251.254:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:10.361483 25496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:10.362895 25496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:10.376859 25506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:10.376822 25507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:10.376889 25509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:10.378118 25496 server_base.cc:1034] running on GCE node
I20260808 04:02:10.378841 25496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:10.379968 25496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:10.381208 25496 hybrid_clock.cc:648] HybridClock initialized: now 1786161730381144 us; error 61 us; skew 500 ppm
I20260808 04:02:10.381598 25496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:10 dist-test-slave-pww0 krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161730, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:10.384917 25496 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 04:02:10.386608 25496 webserver.cc:460] Webserver started at http://127.28.251.195:45759/ using document root <none> and password file <none>
I20260808 04:02:10.387355 25496 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:10.387526 25496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:10.393522 25496 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:02:10.396663 25522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:10.398234 25496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:02:10.398422 25496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "03148b5891fe444dac3625203b23adb9"
format_stamp: "Formatted at 2026-08-08 04:01:24 on dist-test-slave-pww0"
server_key: "50eba8b20b7ca7e8c4233dbe2b2123c7"
server_key_iv: "b179bd56d6ee544c7ae2c5e83563fa65"
server_key_version: "encryptionkey@0"
I20260808 04:02:10.399510 25496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:10.412953 25496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:10.417234 25496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:10.417524 25496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:10.418565 25496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:10.420043 25496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:10.420145 25496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:10.420258 25496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:10.420336 25496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:10.453639 25496 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:36103
I20260808 04:02:10.453745 25715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:36103 every 8 connection(s)
I20260808 04:02:10.455283 25496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:02:10.459789 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 25496
I20260808 04:02:10.461187 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:10.458677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:57037)
0808 04:02:10.459689 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:10.460051 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":348,"threads_started":1}
W20260808 04:02:10.462112 25716 heartbeater.cc:644] Failed to heartbeat to 127.28.251.254:41725 (0 consecutive failures): Network error: Failed to ping master at 127.28.251.254:41725: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
I20260808 04:02:10.463089 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:10.462727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:57589)
0808 04:02:10.462887 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:10.462964 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 04:02:10.464046 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:10.463709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:45749)
0808 04:02:10.463844 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:10.463925 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:02:10.464310 25716 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:02:10.491936 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:10.490915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:46139)
0808 04:02:10.491568 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:10.491687 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":304,"threads_started":1}
I20260808 04:02:10.980178 25724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:10.979252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:59663)
0808 04:02:10.979817 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:10.979923 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":287,"threads_started":1}
I20260808 04:02:11.047669 25725 raft_consensus.cc:482] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:11.047991 25725 raft_consensus.cc:504] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:02:11.049695 25725 leader_election.cc:288] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723), 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103)
Aug 08 04:02:11 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
Aug 08 04:02:11 dist-test-slave-pww0 krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 04:02:11.077875 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:11.050838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:36103 (local address 127.28.251.193:56141)
0808 04:02:11.051635 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:11.051656 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:11.051838 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:11.055207 (+ 3369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:11.055219 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:11.056364 (+ 1145us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:11.057573 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:11.057606 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:11.060178 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:11.060182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:11.062778 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:11.062802 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:11.064423 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:11.064431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:11.064790 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:11.065748 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:02:11.065792 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:11.069658 (+ 3866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:11.073494 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:11.073507 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:11.073524 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:11.074167 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:11.074686 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:11.074692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:11.074696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:11.075003 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:11.076054 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:11.076061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:11.076404 (+ 343us) client_negotiation.cc:769] Sending connection context
0808 04:02:11.076754 (+ 350us) client_negotiation.cc:240] Negotiation successful
0808 04:02:11.077335 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":421,"spinlock_wait_cycles":22400,"thread_start_us":462,"threads_started":1}
I20260808 04:02:11.077975 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:11.049988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:42723 (local address 127.28.251.193:43415)
0808 04:02:11.050631 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:11.050672 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:11.051358 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:11.055207 (+ 3849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:11.055219 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:11.056364 (+ 1145us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:11.057573 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:11.057606 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:11.060164 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:11.060175 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:11.062778 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:11.062802 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:11.064392 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:11.064403 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:11.064798 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:11.065748 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:02:11.065821 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:11.069658 (+ 3837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:11.073563 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:11.073567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:11.073570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:11.074152 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:11.074740 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:11.074748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:11.074751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:11.075007 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:11.076810 (+ 1803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:11.076815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:11.076966 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 04:02:11.077105 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 04:02:11.077516 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":391,"spinlock_wait_cycles":48128,"thread_start_us":382,"threads_started":1}
I20260808 04:02:11.077925 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:11.051384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:56141 (local address 127.28.251.195:36103)
0808 04:02:11.052147 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:11.052161 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:11.052202 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:02:11.054319 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:11.054331 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:11.054726 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:11.055046 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:11.057857 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:11.059889 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:11.063000 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:11.064158 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:11.064392 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:11.070041 (+ 5649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:11.070050 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:11.070067 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:11.070114 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:11.073292 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:11.074341 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:11.074349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:11.074355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:11.074550 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:11.075163 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:11.075168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:11.075170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:11.075875 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:11.076035 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:11.077063 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0808 04:02:11.077523 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":449,"threads_started":1}
I20260808 04:02:11.078616 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:11.050390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:43415 (local address 127.28.251.194:42723)
0808 04:02:11.051056 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:11.051074 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:11.051129 (+ 55us) server_negotiation.cc:407] Connection header received
0808 04:02:11.054003 (+ 2874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:11.054019 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:11.054573 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:11.055000 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:11.057857 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:11.059889 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:11.062999 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:11.064158 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:11.064392 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:11.070041 (+ 5649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:11.070051 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:11.070067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:11.070114 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:11.073398 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:11.074348 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:11.074358 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:11.074366 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:11.074601 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:11.076103 (+ 1502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:11.076107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:11.076109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:11.076657 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:11.076853 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:11.077399 (+ 546us) server_negotiation.cc:299] Negotiation successful
0808 04:02:11.077862 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":321,"threads_started":1}
W20260808 04:02:11.081688 25069 leader_election.cc:341] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723): Not found: Tablet not found: a8bed27dbaff4a2794bd0263376cbdc7
W20260808 04:02:11.082044 25069 leader_election.cc:341] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103): Not found: Tablet not found: a8bed27dbaff4a2794bd0263376cbdc7
I20260808 04:02:11.082167 25069 leader_election.cc:302] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e161f3da09a342d7afab36246bccebbd; no voters: 03148b5891fe444dac3625203b23adb9, c2956e6c23b842d8be00d1e2c200b612
I20260808 04:02:11.082559 25725 raft_consensus.cc:2726] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:02:11.465898 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:11.464890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:57959)
0808 04:02:11.465455 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:11.465626 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":268,"threads_started":1}
I20260808 04:02:11.493412 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:11.492922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:35853)
0808 04:02:11.493140 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:11.493259 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 04:02:11.982523 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:11.981434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:59205)
0808 04:02:11.982085 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:11.982272 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":354,"threads_started":1}
I20260808 04:02:12.467906 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:12.467051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:59035)
0808 04:02:12.467636 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:12.467739 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":290,"threads_started":1}
I20260808 04:02:12.495620 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:12.494532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:55425)
0808 04:02:12.495281 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:12.495381 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":403,"threads_started":1}
I20260808 04:02:12.984591 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:12.983723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:33935)
0808 04:02:12.984292 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:12.984396 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":294,"threads_started":1}
I20260808 04:02:13.078280 25742 raft_consensus.cc:482] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:13.078516 25742 raft_consensus.cc:504] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:02:13.079304 25742 leader_election.cc:288] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723), 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103)
W20260808 04:02:13.080483 25069 leader_election.cc:341] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723): Not found: Tablet not found: a8bed27dbaff4a2794bd0263376cbdc7
W20260808 04:02:13.080744 25069 leader_election.cc:341] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103): Not found: Tablet not found: a8bed27dbaff4a2794bd0263376cbdc7
I20260808 04:02:13.080862 25069 leader_election.cc:302] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e161f3da09a342d7afab36246bccebbd; no voters: 03148b5891fe444dac3625203b23adb9, c2956e6c23b842d8be00d1e2c200b612
I20260808 04:02:13.081131 25742 raft_consensus.cc:2726] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:02:13.469715 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:13.468914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:42855)
0808 04:02:13.469429 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:13.469531 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":258,"threads_started":1}
I20260808 04:02:13.497480 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:13.496614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:55203)
0808 04:02:13.497124 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:13.497256 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":237,"threads_started":1}
I20260808 04:02:13.986395 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:13.985618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:36539)
0808 04:02:13.986109 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:13.986202 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260808 04:02:14.471736 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:14.470890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:46653)
0808 04:02:14.471400 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:14.471542 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":240,"threads_started":1}
I20260808 04:02:14.499430 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:14.498632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:36023)
0808 04:02:14.499130 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:14.499228 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
I20260808 04:02:14.988466 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:14.987611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:47169)
0808 04:02:14.988184 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:14.988289 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":319,"threads_started":1}
I20260808 04:02:15.473524 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:15.472730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:37117)
0808 04:02:15.473247 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:15.473342 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":275,"threads_started":1}
I20260808 04:02:15.501382 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:15.500499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:38749)
0808 04:02:15.501103 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:15.501197 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":344,"threads_started":1}
I20260808 04:02:15.810303 25757 raft_consensus.cc:482] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:15.810545 25757 raft_consensus.cc:504] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:02:15.811235 25757 leader_election.cc:288] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723), 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103)
W20260808 04:02:15.812311 25069 leader_election.cc:341] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723): Not found: Tablet not found: a8bed27dbaff4a2794bd0263376cbdc7
W20260808 04:02:15.812515 25069 leader_election.cc:341] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103): Not found: Tablet not found: a8bed27dbaff4a2794bd0263376cbdc7
I20260808 04:02:15.812630 25069 leader_election.cc:302] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e161f3da09a342d7afab36246bccebbd; no voters: 03148b5891fe444dac3625203b23adb9, c2956e6c23b842d8be00d1e2c200b612
I20260808 04:02:15.812893 25757 raft_consensus.cc:2726] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:02:15.990399 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:15.989478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:41057)
0808 04:02:15.990070 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:15.990175 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":318,"threads_started":1}
I20260808 04:02:16.475574 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:16.474666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:46957)
0808 04:02:16.475230 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:16.475338 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":302,"threads_started":1}
I20260808 04:02:16.503319 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:16.502434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:43659)
0808 04:02:16.502987 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:16.503111 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":284,"threads_started":1}
I20260808 04:02:16.992473 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:16.991577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:47991)
0808 04:02:16.992151 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:16.992266 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":254,"threads_started":1}
I20260808 04:02:17.477610 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:17.476729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:44687)
0808 04:02:17.477313 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:17.477415 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":251,"threads_started":1}
I20260808 04:02:17.505235 25766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:17.504391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:39639)
0808 04:02:17.504933 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:17.505041 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":248,"threads_started":1}
I20260808 04:02:17.994223 25767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:17.993470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:54763)
0808 04:02:17.993930 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:17.994028 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":221,"threads_started":1}
I20260808 04:02:18.479648 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:18.478710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:44259)
0808 04:02:18.479250 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:18.479356 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":268,"threads_started":1}
I20260808 04:02:18.507056 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:18.506236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:57937)
0808 04:02:18.506788 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:18.506876 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":282,"threads_started":1}
I20260808 04:02:18.996248 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:18.995342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:34517)
0808 04:02:18.995951 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:18.996060 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":295,"threads_started":1}
I20260808 04:02:19.352626 25774 raft_consensus.cc:482] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:19.352841 25774 raft_consensus.cc:504] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e161f3da09a342d7afab36246bccebbd" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 44809 } } peers { permanent_uuid: "c2956e6c23b842d8be00d1e2c200b612" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 42723 } } peers { permanent_uuid: "03148b5891fe444dac3625203b23adb9" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36103 } }
I20260808 04:02:19.353469 25774 leader_election.cc:288] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723), 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103)
W20260808 04:02:19.354532 25069 leader_election.cc:341] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2956e6c23b842d8be00d1e2c200b612 (127.28.251.194:42723): Not found: Tablet not found: a8bed27dbaff4a2794bd0263376cbdc7
W20260808 04:02:19.354764 25069 leader_election.cc:341] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03148b5891fe444dac3625203b23adb9 (127.28.251.195:36103): Not found: Tablet not found: a8bed27dbaff4a2794bd0263376cbdc7
I20260808 04:02:19.354847 25069 leader_election.cc:302] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e161f3da09a342d7afab36246bccebbd; no voters: 03148b5891fe444dac3625203b23adb9, c2956e6c23b842d8be00d1e2c200b612
I20260808 04:02:19.355100 25774 raft_consensus.cc:2726] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:02:19.481523 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:19.480708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:40799)
0808 04:02:19.481261 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:19.481355 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":243,"threads_started":1}
I20260808 04:02:19.509172 25781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:19.508253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:38823)
0808 04:02:19.508819 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:19.508931 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":268,"threads_started":1}
I20260808 04:02:19.998136 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:19.997317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:47099)
0808 04:02:19.997829 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:19.997933 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":245,"threads_started":1}
I20260808 04:02:20.483434 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:20.482533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:55251)
0808 04:02:20.483066 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:20.483162 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":276,"threads_started":1}
I20260808 04:02:20.511119 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:20.510274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:44167)
0808 04:02:20.510831 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:20.510928 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":277,"threads_started":1}
I20260808 04:02:21.000105 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:20.999229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:57323)
0808 04:02:20.999827 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:20.999923 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":301,"threads_started":1}
I20260808 04:02:21.485308 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:21.484454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:42243)
0808 04:02:21.485031 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:21.485137 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":298,"threads_started":1}
I20260808 04:02:21.512975 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:21.512125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:35435)
0808 04:02:21.512657 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:21.512797 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":278,"threads_started":1}
I20260808 04:02:22.001852 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:22.001068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:34659)
0808 04:02:22.001544 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:22.001636 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":242,"threads_started":1}
I20260808 04:02:22.487149 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:22.486303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:35931)
0808 04:02:22.486859 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:22.486955 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":267,"threads_started":1}
I20260808 04:02:22.514732 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:22.513942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:52883)
0808 04:02:22.514420 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:22.514523 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":233,"threads_started":1}
I20260808 04:02:23.003887 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:23.002890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:42533)
0808 04:02:23.003521 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:23.003656 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":263,"threads_started":1}
I20260808 04:02:23.489275 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:23.488274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:60925)
0808 04:02:23.488917 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:23.489054 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":292,"threads_started":1}
I20260808 04:02:23.516626 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:23.515818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:41709)
0808 04:02:23.516326 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:23.516420 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":238,"threads_started":1}
I20260808 04:02:24.005723 25808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:24.004875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:45491)
0808 04:02:24.005409 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:24.005509 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":238,"threads_started":1}
I20260808 04:02:24.491147 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:24.490303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:42023)
0808 04:02:24.490835 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:24.490937 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":243,"threads_started":1}
I20260808 04:02:24.518985 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:24.517954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.193:50859)
0808 04:02:24.518566 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:24.518729 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":270,"threads_started":1}
I20260808 04:02:25.007974 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:25.006910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.194:53933)
0808 04:02:25.007569 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:25.007739 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":250,"threads_started":1}
I20260808 04:02:25.469857 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 24969
I20260808 04:02:25.487214 25817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:25.471387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.28.251.193:44809)
0808 04:02:25.472565 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:25.472575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:25.472609 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:02:25.472681 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:25.472689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:25.472910 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:25.473054 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:25.474600 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:25.475853 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:25.477240 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:25.477828 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:25.477927 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:25.480716 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:25.480731 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:25.480749 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:25.480816 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:25.484057 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:25.484966 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:25.484975 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:25.484978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:25.485144 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:25.485544 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:25.485549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:25.485551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:25.485987 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:25.486244 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:25.486577 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 04:02:25.486874 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":785,"threads_started":1}
I20260808 04:02:25.488634 25241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:25.494690 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:25.492450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41725 (local address 127.28.251.195:60119)
0808 04:02:25.494360 (+ 1910us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:25.494512 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.254:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1783,"thread_start_us":1630,"threads_started":1}
I20260808 04:02:25.582576 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 24969
I20260808 04:02:25.582540 25822 raft_consensus.cc:482] T a8bed27dbaff4a2794bd0263376cbdc7 P e161f3da09a342d7afab36246bccebbd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:25.614557 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 25261
I20260808 04:02:25.628780 25827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:25.615358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.28.251.194:42723)
0808 04:02:25.615846 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:25.615851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:25.615868 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:02:25.615927 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:25.615931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:25.615999 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:25.616093 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:25.617505 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:25.618305 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:25.619597 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:25.620144 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:25.620268 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:25.622971 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:25.622977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:25.622980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:25.623020 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:25.625791 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:25.626462 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:25.626467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:25.626469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:25.626639 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:25.627015 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:25.627020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:25.627022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:25.627378 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:25.627720 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:25.628095 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 04:02:25.628305 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
I20260808 04:02:25.629606 25485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:25.715233 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 25261
I20260808 04:02:25.741572 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 25496
I20260808 04:02:25.756685 25829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:25.742555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.28.251.195:36103)
0808 04:02:25.743197 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:25.743202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:25.743220 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:02:25.743280 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:25.743284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:25.743360 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:25.743494 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:25.745046 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:25.745867 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:25.747362 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:25.747886 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:25.748015 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:25.750828 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:25.750835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:25.750838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:25.750882 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:25.753705 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:25.754466 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:25.754471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:25.754473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:25.754639 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:25.755062 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:25.755066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:25.755068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:25.755364 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:25.755694 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:25.756038 (+ 344us) server_negotiation.cc:299] Negotiation successful
0808 04:02:25.756294 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":324,"threads_started":1}
I20260808 04:02:25.757457 25706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:25.844915 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 25496
2026-08-08T04:02:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65574 ms)
[----------] 30 tests from SecurityITest (315432 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:02:25 dist-test-slave-pww0 krb5kdc[25837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:02:25 dist-test-slave-pww0 krb5kdc[25837](info): set up 2 sockets
Aug 08 04:02:25 dist-test-slave-pww0 krb5kdc[25837](info): commencing operation
krb5kdc: starting...
W20260808 04:02:27.142544 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:02:27 dist-test-slave-pww0 krb5kdc[25837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:02:27.511890 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40543
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:40543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:27.879005 25858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:27.879370 25858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:27.879545 25858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:27.887084 25858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:02:27.887234 25858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:27.887326 25858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:27.887403 25858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:02:27.887485 25858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:02:27.898025 25858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:40543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:27.899490 25858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:27.900889 25858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:27.914093 25864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:27.914155 25865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:27.914419 25858 server_base.cc:1034] running on GCE node
W20260808 04:02:27.914125 25871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:27.915537 25858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:27.917121 25858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:27.918336 25858 hybrid_clock.cc:648] HybridClock initialized: now 1786161747918277 us; error 47 us; skew 500 ppm
I20260808 04:02:27.918740 25858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:27 dist-test-slave-pww0 krb5kdc[25837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161747, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:27.921968 25858 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:02:27.923619 25858 webserver.cc:460] Webserver started at http://127.28.251.254:37781/ using document root <none> and password file <none>
I20260808 04:02:27.924360 25858 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:27.924469 25858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:27.924762 25858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:27.928133 25858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "16e44c0e4ea74882bd21dc51c3fbfa8c"
format_stamp: "Formatted at 2026-08-08 04:02:27 on dist-test-slave-pww0"
server_key: "3f6fdabaf89ef528974e55f9ae974f03"
server_key_iv: "1ec97ea3e6ce4175c95cafe178f40cd7"
server_key_version: "encryptionkey@0"
I20260808 04:02:27.928987 25858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "16e44c0e4ea74882bd21dc51c3fbfa8c"
format_stamp: "Formatted at 2026-08-08 04:02:27 on dist-test-slave-pww0"
server_key: "3f6fdabaf89ef528974e55f9ae974f03"
server_key_iv: "1ec97ea3e6ce4175c95cafe178f40cd7"
server_key_version: "encryptionkey@0"
I20260808 04:02:27.934827 25858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:02:27.938769 25881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:27.940279 25858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:02:27.940474 25858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "16e44c0e4ea74882bd21dc51c3fbfa8c"
format_stamp: "Formatted at 2026-08-08 04:02:27 on dist-test-slave-pww0"
server_key: "3f6fdabaf89ef528974e55f9ae974f03"
server_key_iv: "1ec97ea3e6ce4175c95cafe178f40cd7"
server_key_version: "encryptionkey@0"
I20260808 04:02:27.940680 25858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:27.955353 25858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:27.959882 25858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:27.960173 25858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:27.979933 25858 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:40543
I20260808 04:02:27.979965 25957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:40543 every 8 connection(s)
I20260808 04:02:27.981505 25858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:02:27.985747 25958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:27.989257 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 25858
I20260808 04:02:27.989573 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:02:27.990463 29679 external_mini_cluster.cc:1428] Setting key 1545f090d2b4df02bd647fd384bd6529
I20260808 04:02:27.999724 25958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c: Bootstrap starting.
I20260808 04:02:28.003161 25958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:28.004684 25958 log.cc:824] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:02:28 dist-test-slave-pww0 krb5kdc[25837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:28.008529 25958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c: No bootstrap required, opened a new log
I20260808 04:02:28.017828 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:27.993149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.28.251.254:40543)
0808 04:02:27.994060 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:27.994073 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:27.994124 (+ 51us) server_negotiation.cc:407] Connection header received
0808 04:02:27.995994 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:27.996023 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:27.996495 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:27.996931 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:27.998571 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:28.000121 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:28.001746 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:28.002484 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:28.002668 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:28.007877 (+ 5209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:28.007894 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:28.007918 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:28.007978 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:28.012502 (+ 4524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:28.013411 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:28.013422 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:28.013434 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:28.013678 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:28.014083 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:28.014091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:28.014095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:28.014936 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:28.015262 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:28.015690 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 04:02:28.016234 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":394,"threads_started":1}
I20260808 04:02:28.019929 25958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e44c0e4ea74882bd21dc51c3fbfa8c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40543 } }
I20260808 04:02:28.020334 25958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:28.020443 25958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e44c0e4ea74882bd21dc51c3fbfa8c, State: Initialized, Role: FOLLOWER
I20260808 04:02:28.021052 25958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e44c0e4ea74882bd21dc51c3fbfa8c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40543 } }
I20260808 04:02:28.021327 25958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:02:28.021482 25958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:02:28.021654 25958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:28.024050 25958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e44c0e4ea74882bd21dc51c3fbfa8c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40543 } }
I20260808 04:02:28.024606 25958 leader_election.cc:302] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16e44c0e4ea74882bd21dc51c3fbfa8c; no voters:
I20260808 04:02:28.025254 25958 leader_election.cc:288] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:02:28.025405 25969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:28.026551 25969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [term 1 LEADER]: Becoming Leader. State: Replica: 16e44c0e4ea74882bd21dc51c3fbfa8c, State: Running, Role: LEADER
I20260808 04:02:28.027220 25969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e44c0e4ea74882bd21dc51c3fbfa8c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40543 } }
I20260808 04:02:28.027583 25958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:02:28.031641 25970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16e44c0e4ea74882bd21dc51c3fbfa8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e44c0e4ea74882bd21dc51c3fbfa8c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40543 } } }
I20260808 04:02:28.032092 25970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:28.031953 25971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e44c0e4ea74882bd21dc51c3fbfa8c. Latest consensus state: current_term: 1 leader_uuid: "16e44c0e4ea74882bd21dc51c3fbfa8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e44c0e4ea74882bd21dc51c3fbfa8c" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40543 } } }
I20260808 04:02:28.032263 25971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:28.034816 25983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:02:28.040583 25983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:02:28.050688 25983 catalog_manager.cc:1347] Generated new cluster ID: de99356e08a9413f9fb46320d75986a2
I20260808 04:02:28.050835 25983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:02:28.064699 25983 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:02:28.066378 25983 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:02:28.077186 25983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c: Generated new TSK 0
I20260808 04:02:28.078186 25983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 04:02:28.174774 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40543
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:28.547820 26009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:28.548154 26009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:28.548280 26009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:28.556176 26009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:28.556329 26009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:28.556501 26009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:02:28.567111 26009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:28.568845 26009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:28.570390 26009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:28.585515 26015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:28.585438 26016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:28.585772 26018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:28.586162 26009 server_base.cc:1034] running on GCE node
I20260808 04:02:28.587021 26009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:28.588186 26009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:28.589409 26009 hybrid_clock.cc:648] HybridClock initialized: now 1786161748589347 us; error 51 us; skew 500 ppm
I20260808 04:02:28.589792 26009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:28 dist-test-slave-pww0 krb5kdc[25837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161748, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:28.598234 26009 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 04:02:28.600034 26009 webserver.cc:460] Webserver started at http://127.28.251.193:40841/ using document root <none> and password file <none>
I20260808 04:02:28.600872 26009 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:28.601019 26009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:28.601368 26009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:28.604750 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "e74115551f804886b526b17f5d35f717"
format_stamp: "Formatted at 2026-08-08 04:02:28 on dist-test-slave-pww0"
server_key: "a02e86f9b5cf4ea9a89b69a2f87754b3"
server_key_iv: "db6fadbddc5cb01b743f662c37533590"
server_key_version: "encryptionkey@0"
I20260808 04:02:28.605581 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "e74115551f804886b526b17f5d35f717"
format_stamp: "Formatted at 2026-08-08 04:02:28 on dist-test-slave-pww0"
server_key: "a02e86f9b5cf4ea9a89b69a2f87754b3"
server_key_iv: "db6fadbddc5cb01b743f662c37533590"
server_key_version: "encryptionkey@0"
I20260808 04:02:28.611750 26009 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260808 04:02:28.615860 26029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:28.617448 26009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:02:28.617666 26009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "e74115551f804886b526b17f5d35f717"
format_stamp: "Formatted at 2026-08-08 04:02:28 on dist-test-slave-pww0"
server_key: "a02e86f9b5cf4ea9a89b69a2f87754b3"
server_key_iv: "db6fadbddc5cb01b743f662c37533590"
server_key_version: "encryptionkey@0"
I20260808 04:02:28.617902 26009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:28.637282 26009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:28.642010 26009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:28.642341 26009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:28.643436 26009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:28.645151 26009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:28.645269 26009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:28.645411 26009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:28.645469 26009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:28.679121 26009 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:33743
I20260808 04:02:28.679172 26222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:33743 every 8 connection(s)
I20260808 04:02:28.680542 26009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:02:28.682613 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 26009
I20260808 04:02:28.682848 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 04:02:28.683771 29679 external_mini_cluster.cc:1428] Setting key 8a04acd39fe5648382b14388d25d7e99
Aug 08 04:02:28 dist-test-slave-pww0 krb5kdc[25837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161748, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:28.703590 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:28.683493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:59923 (local address 127.28.251.254:40543)
0808 04:02:28.683997 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:28.684002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:28.685369 (+ 1367us) server_negotiation.cc:407] Connection header received
0808 04:02:28.687690 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:28.687696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:28.687798 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:28.687915 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:28.689907 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:28.690894 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:28.692684 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:28.693230 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:28.693365 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:28.697390 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:28.697396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:28.697400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:28.697439 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:28.699927 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:28.700878 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:28.700882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:28.700885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:28.701018 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:28.701532 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:28.701536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:28.701538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:28.701836 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:28.701967 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:28.702906 (+ 939us) server_negotiation.cc:299] Negotiation successful
0808 04:02:28.703163 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":245,"threads_started":1}
I20260808 04:02:28.704435 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:28.684234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40543 (local address 127.28.251.193:59923)
0808 04:02:28.685160 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:28.685215 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:28.687341 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:28.688232 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:28.688243 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:28.688927 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:28.689680 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:28.689705 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:28.691092 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:28.691096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:28.692511 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:28.692526 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:28.693494 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:28.693500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:28.693742 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:28.694447 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:02:28.694478 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:28.697052 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:28.700089 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:28.700099 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:28.700119 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:28.700731 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:28.701124 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:28.701130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:28.701133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:28.701404 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:28.701963 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:28.701971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:28.702451 (+ 480us) client_negotiation.cc:769] Sending connection context
0808 04:02:28.702715 (+ 264us) client_negotiation.cc:240] Negotiation successful
0808 04:02:28.703153 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":359,"threads_started":1}
I20260808 04:02:28.706233 26224 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40543
I20260808 04:02:28.706634 26224 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:28.707687 26224 heartbeater.cc:511] Master 127.28.251.254:40543 requested a full tablet report, sending...
I20260808 04:02:28.710659 25904 ts_manager.cc:194] Registered new tserver with Master: e74115551f804886b526b17f5d35f717 (127.28.251.193:33743)
I20260808 04:02:28.712838 25904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:59923
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 04:02:28.767313 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40543
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:29.144306 26234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:29.144685 26234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:29.144838 26234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:29.152693 26234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:29.152858 26234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:29.153036 26234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:02:29.163542 26234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:29.165014 26234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:29.166448 26234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:29.180694 26249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:29.180676 26246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:29.181268 26243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:29.185139 26234 server_base.cc:1034] running on GCE node
I20260808 04:02:29.185846 26234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:29.186901 26234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:29.188143 26234 hybrid_clock.cc:648] HybridClock initialized: now 1786161749188077 us; error 47 us; skew 500 ppm
I20260808 04:02:29.188512 26234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:29 dist-test-slave-pww0 krb5kdc[25837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161749, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:29.191882 26234 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 04:02:29.193579 26234 webserver.cc:460] Webserver started at http://127.28.251.194:43869/ using document root <none> and password file <none>
I20260808 04:02:29.194379 26234 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:29.194514 26234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:29.194859 26234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:29.198112 26234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5"
format_stamp: "Formatted at 2026-08-08 04:02:29 on dist-test-slave-pww0"
server_key: "9d169e59a5dbf48aac3c5cec6e0c2f7d"
server_key_iv: "453fa7f222ba5d7d9add38c0ed86700f"
server_key_version: "encryptionkey@0"
I20260808 04:02:29.198947 26234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5"
format_stamp: "Formatted at 2026-08-08 04:02:29 on dist-test-slave-pww0"
server_key: "9d169e59a5dbf48aac3c5cec6e0c2f7d"
server_key_iv: "453fa7f222ba5d7d9add38c0ed86700f"
server_key_version: "encryptionkey@0"
I20260808 04:02:29.204845 26234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260808 04:02:29.208825 26259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:29.210300 26234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 04:02:29.210493 26234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5"
format_stamp: "Formatted at 2026-08-08 04:02:29 on dist-test-slave-pww0"
server_key: "9d169e59a5dbf48aac3c5cec6e0c2f7d"
server_key_iv: "453fa7f222ba5d7d9add38c0ed86700f"
server_key_version: "encryptionkey@0"
I20260808 04:02:29.210722 26234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:29.242935 26234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:29.247702 26234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:29.248018 26234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:29.249116 26234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:29.250629 26234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:29.250712 26234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:29.250856 26234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:29.250910 26234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:29.284798 26234 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:43463
I20260808 04:02:29.284849 26454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:43463 every 8 connection(s)
I20260808 04:02:29.286201 26234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:02:29.296535 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 26234
I20260808 04:02:29.296746 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 04:02:29.297348 29679 external_mini_cluster.cc:1428] Setting key b73cb4738ff1dea0861676c644260557
Aug 08 04:02:29 dist-test-slave-pww0 krb5kdc[25837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161749, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:29.309017 26457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:29.289090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:50907 (local address 127.28.251.254:40543)
0808 04:02:29.289567 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:29.289571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:29.290565 (+ 994us) server_negotiation.cc:407] Connection header received
0808 04:02:29.292547 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:29.292553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:29.292639 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:29.292741 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:29.294416 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.295166 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.297290 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.297782 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.297940 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:29.302391 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:29.302397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:29.302400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:29.302452 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:29.305281 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:29.306327 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:29.306332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:29.306334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:29.306482 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:29.307041 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:29.307045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:29.307048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:29.307345 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:29.307519 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:29.308480 (+ 961us) server_negotiation.cc:299] Negotiation successful
0808 04:02:29.308710 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":256,"threads_started":1}
I20260808 04:02:29.310614 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:29.289586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40543 (local address 127.28.251.194:50907)
0808 04:02:29.290356 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:29.290403 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:29.292248 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:29.293019 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:29.293028 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:29.293606 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:29.294216 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:29.294235 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.295341 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.295345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:29.297107 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:29.297125 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.298057 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.298063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:29.298356 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:29.299092 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:02:29.299135 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:29.302079 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:29.305475 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:29.305487 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:29.305508 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:29.306173 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:29.306599 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:29.306603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:29.306606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:29.306905 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:29.307515 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:29.307522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:29.308038 (+ 516us) client_negotiation.cc:769] Sending connection context
0808 04:02:29.308367 (+ 329us) client_negotiation.cc:240] Negotiation successful
0808 04:02:29.308981 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":315,"threads_started":1}
I20260808 04:02:29.312321 26455 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40543
I20260808 04:02:29.312670 26455 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:29.313480 26455 heartbeater.cc:511] Master 127.28.251.254:40543 requested a full tablet report, sending...
I20260808 04:02:29.315085 25904 ts_manager.cc:194] Registered new tserver with Master: a1f6341d7be7463f9c3d9fcfc5313fb5 (127.28.251.194:43463)
I20260808 04:02:29.316036 25904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:50907
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 04:02:29.376940 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40543
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:02:29.717272 26224 heartbeater.cc:503] Master 127.28.251.254:40543 was elected leader, sending a full tablet report...
W20260808 04:02:29.749420 26464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:29.749766 26464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:29.749908 26464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:29.757614 26464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:29.757783 26464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:29.757956 26464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:02:29.768497 26464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:29.769992 26464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:29.771559 26464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:29.786164 26473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:29.786114 26474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:29.786197 26478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:29.787299 26464 server_base.cc:1034] running on GCE node
I20260808 04:02:29.788129 26464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:29.789194 26464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:29.790424 26464 hybrid_clock.cc:648] HybridClock initialized: now 1786161749790366 us; error 46 us; skew 500 ppm
I20260808 04:02:29.790763 26464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:29 dist-test-slave-pww0 krb5kdc[25837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161749, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:29.794176 26464 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 04:02:29.795936 26464 webserver.cc:460] Webserver started at http://127.28.251.195:43891/ using document root <none> and password file <none>
I20260808 04:02:29.796684 26464 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:29.796799 26464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:29.797097 26464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:29.800417 26464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "5e5108a4a9d34a6899aecb645257d921"
format_stamp: "Formatted at 2026-08-08 04:02:29 on dist-test-slave-pww0"
server_key: "b71bb8548158a5d60fc109db7dde373c"
server_key_iv: "2a64ab8b5be0f8b64a96cd5c835ed5e5"
server_key_version: "encryptionkey@0"
I20260808 04:02:29.801268 26464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "5e5108a4a9d34a6899aecb645257d921"
format_stamp: "Formatted at 2026-08-08 04:02:29 on dist-test-slave-pww0"
server_key: "b71bb8548158a5d60fc109db7dde373c"
server_key_iv: "2a64ab8b5be0f8b64a96cd5c835ed5e5"
server_key_version: "encryptionkey@0"
I20260808 04:02:29.807180 26464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 04:02:29.811993 26490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:29.813503 26464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:02:29.813687 26464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "5e5108a4a9d34a6899aecb645257d921"
format_stamp: "Formatted at 2026-08-08 04:02:29 on dist-test-slave-pww0"
server_key: "b71bb8548158a5d60fc109db7dde373c"
server_key_iv: "2a64ab8b5be0f8b64a96cd5c835ed5e5"
server_key_version: "encryptionkey@0"
I20260808 04:02:29.813906 26464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:29.839612 26464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:29.844333 26464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:29.844699 26464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:29.845795 26464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:29.847378 26464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:29.847481 26464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:29.847607 26464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:29.847656 26464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:29.882054 26464 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:35513
I20260808 04:02:29.882107 26684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:35513 every 8 connection(s)
I20260808 04:02:29.883435 26464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:02:29.884122 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 26464
I20260808 04:02:29.884305 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 04:02:29.884919 29679 external_mini_cluster.cc:1428] Setting key 9d31927eab728ffc25eb23f157f41d16
Aug 08 04:02:29 dist-test-slave-pww0 krb5kdc[25837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161749, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:29.905441 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:29.886271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:36265 (local address 127.28.251.254:40543)
0808 04:02:29.886739 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:29.886744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:29.887853 (+ 1109us) server_negotiation.cc:407] Connection header received
0808 04:02:29.890223 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:29.890228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:29.890322 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:29.890433 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:29.892193 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.892993 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.894634 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.895057 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.895175 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:29.898961 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:29.898966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:29.898970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:29.899006 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:29.902057 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:29.902860 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:29.902866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:29.902869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:29.903009 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:29.903517 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:29.903521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:29.903523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:29.903821 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:29.903973 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:29.904774 (+ 801us) server_negotiation.cc:299] Negotiation successful
0808 04:02:29.905030 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":234,"threads_started":1}
I20260808 04:02:29.906502 26689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:29.886888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40543 (local address 127.28.251.195:36265)
0808 04:02:29.887687 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:29.887746 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:29.889931 (+ 2185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:29.890712 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:29.890720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:29.891271 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:29.892010 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:29.892034 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.893158 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.893163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:29.894463 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:29.894481 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.895320 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.895334 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:29.895665 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:29.896375 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:02:29.896401 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:29.898703 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:29.902208 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:29.902218 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:29.902236 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:29.902717 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:29.903126 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:29.903131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:29.903134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:29.903361 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:29.903967 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:29.903976 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:29.904367 (+ 391us) client_negotiation.cc:769] Sending connection context
0808 04:02:29.904626 (+ 259us) client_negotiation.cc:240] Negotiation successful
0808 04:02:29.905246 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":607,"threads_started":1}
I20260808 04:02:29.908102 26685 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40543
I20260808 04:02:29.908452 26685 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:29.909312 26685 heartbeater.cc:511] Master 127.28.251.254:40543 requested a full tablet report, sending...
I20260808 04:02:29.911087 25904 ts_manager.cc:194] Registered new tserver with Master: 5e5108a4a9d34a6899aecb645257d921 (127.28.251.195:35513)
I20260808 04:02:29.912035 25904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:36265
I20260808 04:02:29.914821 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:02:29.933594 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:29.919877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.28.251.254:40543)
0808 04:02:29.920050 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:29.920054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:29.920571 (+ 517us) server_negotiation.cc:407] Connection header received
0808 04:02:29.920774 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:29.920778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:29.920853 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:29.920930 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:29.922499 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.923343 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.924811 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.925225 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.925340 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:29.928203 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:29.928208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:29.928212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:29.928251 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:29.930982 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:29.931748 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:29.931752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:29.931754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:29.931889 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:29.932248 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:29.932251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:29.932253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:29.932640 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:29.932857 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:29.933151 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 04:02:29.933352 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:02:29.937861 25904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:02:29.945230 25904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:02:29.973649 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:29.959348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.28.251.193:33743)
0808 04:02:29.960485 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:29.960495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:29.961214 (+ 719us) server_negotiation.cc:407] Connection header received
0808 04:02:29.961615 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:29.961624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:29.961907 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:29.963528 (+ 1621us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:29.964693 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.967772 (+ 3079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.970779 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.972424 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.972629 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:29.972856 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:29.973103 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 04:02:29.973308 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":778,"threads_started":1}
I20260808 04:02:29.973690 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:29.959141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33743 (local address 127.0.0.1:52454)
0808 04:02:29.961041 (+ 1900us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:29.961092 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:29.961377 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:29.963942 (+ 2565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:29.963953 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:29.963991 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:29.964461 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:29.964476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.967977 (+ 3501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.967983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:29.970548 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:29.970566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.972636 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.972642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:29.972845 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:29.972905 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 04:02:29.973030 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 04:02:29.973259 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1619,"mutex_wait_us":1,"thread_start_us":1436,"threads_started":1}
I20260808 04:02:29.974951 26716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:29.961614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:43463 (local address 127.0.0.1:48694)
0808 04:02:29.966006 (+ 4392us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:29.966022 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:29.966451 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:29.967865 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:29.967869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:29.967894 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:29.968307 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:29.968316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.969723 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.969729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:29.972094 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:29.972112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.974081 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.974088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:29.974276 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:29.974319 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 04:02:29.974391 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:02:29.974532 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4281,"thread_start_us":4939,"threads_started":1}
I20260808 04:02:29.975257 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:29.959718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:35513 (local address 127.0.0.1:58654)
0808 04:02:29.965722 (+ 6004us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:29.965746 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:29.965925 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:29.967108 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:29.967114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:29.967138 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:29.967571 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:29.967585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.969668 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.969673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:29.971899 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:29.971916 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.974210 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.974216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:29.974387 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:29.974415 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:02:29.974470 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 04:02:29.974579 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5858,"mutex_wait_us":4,"thread_start_us":5687,"threads_started":1}
I20260808 04:02:29.976028 26712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:29.960007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58654 (local address 127.28.251.195:35513)
0808 04:02:29.960957 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:29.960965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:29.966085 (+ 5120us) server_negotiation.cc:407] Connection header received
0808 04:02:29.966215 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:29.966223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:29.966529 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:29.966745 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:29.967800 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.969501 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.972120 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.973607 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.974756 (+ 1149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:29.974968 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:29.975281 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 04:02:29.975681 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":1104,"threads_started":1}
I20260808 04:02:29.976681 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:29.961947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48694 (local address 127.28.251.194:43463)
0808 04:02:29.967181 (+ 5234us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:29.967191 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:29.967221 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:02:29.967340 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:29.967349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:29.967648 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:29.967856 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:29.968479 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.969545 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.972296 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:29.973826 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:29.975282 (+ 1456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:29.975501 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:29.975586 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 04:02:29.976337 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5047,"thread_start_us":4896,"threads_started":1}
I20260808 04:02:29.978176 26106 tablet_service.cc:1467] Processing CreateTablet for tablet 23a3fc0f29cf4c8da0a763c6cadb4170 (DEFAULT_TABLE table=test-table [id=d6c90182728f44d9bee4ab3be4fa41ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:29.979981 26106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a3fc0f29cf4c8da0a763c6cadb4170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:29.980155 26573 tablet_service.cc:1467] Processing CreateTablet for tablet 23a3fc0f29cf4c8da0a763c6cadb4170 (DEFAULT_TABLE table=test-table [id=d6c90182728f44d9bee4ab3be4fa41ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:29.981123 26348 tablet_service.cc:1467] Processing CreateTablet for tablet 23a3fc0f29cf4c8da0a763c6cadb4170 (DEFAULT_TABLE table=test-table [id=d6c90182728f44d9bee4ab3be4fa41ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:29.981734 26573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a3fc0f29cf4c8da0a763c6cadb4170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:29.983016 26348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a3fc0f29cf4c8da0a763c6cadb4170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:30.002187 26723 tablet_bootstrap.cc:492] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: Bootstrap starting.
I20260808 04:02:30.006155 26723 tablet_bootstrap.cc:654] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:30.008317 26719 tablet_bootstrap.cc:492] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: Bootstrap starting.
I20260808 04:02:30.008316 26724 tablet_bootstrap.cc:492] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: Bootstrap starting.
I20260808 04:02:30.011711 26723 log.cc:824] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:30.012217 26724 tablet_bootstrap.cc:654] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:30.012279 26719 tablet_bootstrap.cc:654] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:30.013672 26719 log.cc:824] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:30.014901 26724 log.cc:824] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:30.015719 26723 tablet_bootstrap.cc:492] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: No bootstrap required, opened a new log
I20260808 04:02:30.016052 26723 ts_tablet_manager.cc:1397] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260808 04:02:30.017143 26719 tablet_bootstrap.cc:492] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: No bootstrap required, opened a new log
I20260808 04:02:30.017489 26719 ts_tablet_manager.cc:1397] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 04:02:30.017923 26724 tablet_bootstrap.cc:492] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: No bootstrap required, opened a new log
I20260808 04:02:30.018198 26724 ts_tablet_manager.cc:1397] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260808 04:02:30.026096 26724 raft_consensus.cc:348] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } }
I20260808 04:02:30.027108 26724 raft_consensus.cc:374] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:30.027266 26724 raft_consensus.cc:729] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f6341d7be7463f9c3d9fcfc5313fb5, State: Initialized, Role: FOLLOWER
I20260808 04:02:30.027525 26723 raft_consensus.cc:348] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } }
I20260808 04:02:30.028019 26723 raft_consensus.cc:374] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:30.028200 26723 raft_consensus.cc:729] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5108a4a9d34a6899aecb645257d921, State: Initialized, Role: FOLLOWER
I20260808 04:02:30.028115 26724 consensus_queue.cc:260] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } }
I20260808 04:02:30.028431 26719 raft_consensus.cc:348] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } }
I20260808 04:02:30.028890 26719 raft_consensus.cc:374] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:30.028992 26719 raft_consensus.cc:729] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74115551f804886b526b17f5d35f717, State: Initialized, Role: FOLLOWER
I20260808 04:02:30.028865 26723 consensus_queue.cc:260] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } }
I20260808 04:02:30.029608 26719 consensus_queue.cc:260] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } }
I20260808 04:02:30.030290 26685 heartbeater.cc:503] Master 127.28.251.254:40543 was elected leader, sending a full tablet report...
I20260808 04:02:30.031010 26723 ts_tablet_manager.cc:1428] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260808 04:02:30.032383 26455 heartbeater.cc:503] Master 127.28.251.254:40543 was elected leader, sending a full tablet report...
I20260808 04:02:30.034343 26719 ts_tablet_manager.cc:1428] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
I20260808 04:02:30.035821 26724 ts_tablet_manager.cc:1428] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: Time spent starting tablet: real 0.018s user 0.009s sys 0.007s
W20260808 04:02:30.038338 26456 tablet.cc:2369] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:02:30.135488 26687 tablet.cc:2369] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:02:30.183378 26225 tablet.cc:2369] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:02:30.480400 26737 raft_consensus.cc:482] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:30.480728 26737 raft_consensus.cc:504] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } }
I20260808 04:02:30.482384 26737 leader_election.cc:288] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e74115551f804886b526b17f5d35f717 (127.28.251.193:33743), a1f6341d7be7463f9c3d9fcfc5313fb5 (127.28.251.194:43463)
I20260808 04:02:30.485473 26735 raft_consensus.cc:482] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:30.485945 26735 raft_consensus.cc:504] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } }
I20260808 04:02:30.488773 26740 raft_consensus.cc:482] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:30.489156 26740 raft_consensus.cc:504] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } }
I20260808 04:02:30.490842 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.483425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:43463 (local address 127.28.251.195:42579)
0808 04:02:30.484010 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:30.484030 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:30.484161 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:30.484584 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:30.484586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:30.484607 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:30.484859 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.484867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.486282 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.486287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.487888 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.487904 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.489865 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.489872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.490097 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.490138 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 04:02:30.490241 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 04:02:30.490438 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":231,"threads_started":1}
I20260808 04:02:30.491276 26740 leader_election.cc:288] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1f6341d7be7463f9c3d9fcfc5313fb5 (127.28.251.194:43463), 5e5108a4a9d34a6899aecb645257d921 (127.28.251.195:35513)
I20260808 04:02:30.491735 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.483604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:42579 (local address 127.28.251.194:43463)
0808 04:02:30.484267 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:30.484273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:30.484292 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:02:30.484372 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:30.484377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:30.484457 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:30.484563 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:30.485031 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.486137 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.488071 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.489463 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.489643 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.489727 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:30.491085 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0808 04:02:30.491271 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":347,"threads_started":1}
I20260808 04:02:30.492069 26735 leader_election.cc:288] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e74115551f804886b526b17f5d35f717 (127.28.251.193:33743), 5e5108a4a9d34a6899aecb645257d921 (127.28.251.195:35513)
I20260808 04:02:30.495194 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.482685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33743 (local address 127.28.251.195:44131)
0808 04:02:30.483199 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:30.483220 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:30.483365 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:30.484046 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:30.484051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:30.484084 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:30.484283 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.484294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.485688 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.485695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.487811 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.487828 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.494129 (+ 6301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.494137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.494349 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.494391 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 04:02:30.494459 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 04:02:30.494635 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"spinlock_wait_cycles":2816,"thread_start_us":241,"threads_started":1}
I20260808 04:02:30.495141 26385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a3fc0f29cf4c8da0a763c6cadb4170" candidate_uuid: "5e5108a4a9d34a6899aecb645257d921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" is_pre_election: true
I20260808 04:02:30.495711 26385 raft_consensus.cc:2454] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5108a4a9d34a6899aecb645257d921 in term 0.
I20260808 04:02:30.496240 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.482846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:44131 (local address 127.28.251.193:33743)
0808 04:02:30.483321 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:30.483327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:30.483356 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:02:30.483585 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:30.483592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:30.483686 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:30.483821 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:30.484448 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.485517 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.488018 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.493875 (+ 5857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.495556 (+ 1681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.495645 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:30.495703 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 04:02:30.495869 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
I20260808 04:02:30.496634 26492 leader_election.cc:302] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5108a4a9d34a6899aecb645257d921, a1f6341d7be7463f9c3d9fcfc5313fb5; no voters:
I20260808 04:02:30.497205 26737 raft_consensus.cc:2781] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:02:30.497364 26737 raft_consensus.cc:482] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:02:30.497480 26737 raft_consensus.cc:3037] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:30.501538 26737 raft_consensus.cc:504] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } }
I20260808 04:02:30.502012 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.488532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:35513 (local address 127.28.251.194:57529)
0808 04:02:30.493466 (+ 4934us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:30.493490 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:30.493609 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:30.494471 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:30.494476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:30.494515 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:30.494950 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.494965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.496877 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.496883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.499192 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.499210 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.501138 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.501145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.501331 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.501368 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:02:30.501437 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 04:02:30.501592 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4787,"mutex_wait_us":3,"thread_start_us":3842,"threads_started":1}
I20260808 04:02:30.503175 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.491637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:57529 (local address 127.28.251.195:35513)
0808 04:02:30.494127 (+ 2490us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:30.494132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:30.494156 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:02:30.494213 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:30.494219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:30.494309 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:30.495134 (+ 825us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:30.495504 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.496706 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.499440 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.500864 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.502523 (+ 1659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.502605 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:30.502662 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 04:02:30.502816 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2342,"thread_start_us":2196,"threads_started":1}
I20260808 04:02:30.503779 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.488444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:33743 (local address 127.28.251.194:33513)
0808 04:02:30.492735 (+ 4291us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:30.492756 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:30.492901 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:30.496877 (+ 3976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:30.496883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:30.496911 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:30.497399 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.497413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.499043 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.499049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.501122 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.501136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.502925 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.502932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.503121 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.503156 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:02:30.503229 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 04:02:30.503377 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4141,"mutex_wait_us":1,"thread_start_us":4678,"threads_started":1}
I20260808 04:02:30.504350 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.488930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:33513 (local address 127.28.251.193:33743)
0808 04:02:30.496542 (+ 7612us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:30.496547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:30.496566 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:02:30.496627 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:30.496632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:30.496710 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:30.497553 (+ 843us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:30.497709 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.498862 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.501357 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.502629 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.503662 (+ 1033us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.503751 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:30.503814 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 04:02:30.503945 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":7508,"threads_started":1}
I20260808 04:02:30.504014 26145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a3fc0f29cf4c8da0a763c6cadb4170" candidate_uuid: "5e5108a4a9d34a6899aecb645257d921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74115551f804886b526b17f5d35f717" is_pre_election: true
I20260808 04:02:30.504696 26145 raft_consensus.cc:2454] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5108a4a9d34a6899aecb645257d921 in term 0.
I20260808 04:02:30.504776 26142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a3fc0f29cf4c8da0a763c6cadb4170" candidate_uuid: "5e5108a4a9d34a6899aecb645257d921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74115551f804886b526b17f5d35f717"
I20260808 04:02:30.505103 26142 raft_consensus.cc:3037] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:30.506011 26737 leader_election.cc:288] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [CANDIDATE]: Term 1 election: Requested vote from peers e74115551f804886b526b17f5d35f717 (127.28.251.193:33743), a1f6341d7be7463f9c3d9fcfc5313fb5 (127.28.251.194:43463)
I20260808 04:02:30.507642 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a3fc0f29cf4c8da0a763c6cadb4170" candidate_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5108a4a9d34a6899aecb645257d921" is_pre_election: true
I20260808 04:02:30.508481 26385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a3fc0f29cf4c8da0a763c6cadb4170" candidate_uuid: "5e5108a4a9d34a6899aecb645257d921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5"
I20260808 04:02:30.508630 26602 raft_consensus.cc:2379] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a1f6341d7be7463f9c3d9fcfc5313fb5 in current term 1: Already voted for candidate 5e5108a4a9d34a6899aecb645257d921 in this term.
I20260808 04:02:30.508742 26385 raft_consensus.cc:3037] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:30.509042 26142 raft_consensus.cc:2454] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5108a4a9d34a6899aecb645257d921 in term 1.
I20260808 04:02:30.510366 26492 leader_election.cc:302] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5108a4a9d34a6899aecb645257d921, e74115551f804886b526b17f5d35f717; no voters:
I20260808 04:02:30.510931 26737 raft_consensus.cc:2781] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:30.511034 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.492189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:44507 (local address 127.28.251.194:43463)
0808 04:02:30.492750 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:30.492756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:30.501680 (+ 8924us) server_negotiation.cc:407] Connection header received
0808 04:02:30.502800 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:30.502806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:30.502908 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:30.502975 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:30.504581 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.505601 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.508427 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.509688 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.509870 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.509962 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:30.510614 (+ 652us) server_negotiation.cc:299] Negotiation successful
0808 04:02:30.510769 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446}
I20260808 04:02:30.511107 26737 raft_consensus.cc:686] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 1 LEADER]: Becoming Leader. State: Replica: 5e5108a4a9d34a6899aecb645257d921, State: Running, Role: LEADER
I20260808 04:02:30.511848 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.492144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:43463 (local address 127.28.251.193:44507)
0808 04:02:30.501557 (+ 9413us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:30.501580 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:30.502431 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:30.503202 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:30.503206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:30.503236 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:30.503808 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.503823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.505784 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.505790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.508198 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.508217 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.510136 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.510157 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.510392 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.510431 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 04:02:30.511219 (+ 788us) client_negotiation.cc:240] Negotiation successful
0808 04:02:30.511390 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9239,"mutex_wait_us":3,"thread_start_us":10577,"threads_started":1}
I20260808 04:02:30.511731 26737 consensus_queue.cc:237] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } }
I20260808 04:02:30.512612 26385 raft_consensus.cc:2454] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5108a4a9d34a6899aecb645257d921 in term 1.
I20260808 04:02:30.512650 26383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a3fc0f29cf4c8da0a763c6cadb4170" candidate_uuid: "e74115551f804886b526b17f5d35f717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" is_pre_election: true
I20260808 04:02:30.513113 26383 raft_consensus.cc:2379] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e74115551f804886b526b17f5d35f717 in current term 1: Already voted for candidate 5e5108a4a9d34a6899aecb645257d921 in this term.
I20260808 04:02:30.515362 26142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a3fc0f29cf4c8da0a763c6cadb4170" candidate_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74115551f804886b526b17f5d35f717" is_pre_election: true
I20260808 04:02:30.515801 26142 raft_consensus.cc:2379] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a1f6341d7be7463f9c3d9fcfc5313fb5 in current term 1: Already voted for candidate 5e5108a4a9d34a6899aecb645257d921 in this term.
I20260808 04:02:30.516556 26260 leader_election.cc:302] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1f6341d7be7463f9c3d9fcfc5313fb5; no voters: 5e5108a4a9d34a6899aecb645257d921, e74115551f804886b526b17f5d35f717
I20260808 04:02:30.517092 26735 raft_consensus.cc:2726] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 04:02:30.518864 25901 catalog_manager.cc:5495] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e5108a4a9d34a6899aecb645257d921 (127.28.251.195). New cstate: current_term: 1 leader_uuid: "5e5108a4a9d34a6899aecb645257d921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5108a4a9d34a6899aecb645257d921" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 35513 } health_report { overall_health: HEALTHY } } }
I20260808 04:02:30.522101 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.514709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:51079 (local address 127.28.251.195:35513)
0808 04:02:30.514945 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:30.514950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:30.514969 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:02:30.515037 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:30.515042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:30.515117 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:30.515220 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:30.515957 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.517094 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.520058 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.521067 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.521233 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.521308 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:30.521707 (+ 399us) server_negotiation.cc:299] Negotiation successful
0808 04:02:30.521851 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260808 04:02:30.522102 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.514510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:35513 (local address 127.28.251.193:51079)
0808 04:02:30.514715 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:30.514735 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:30.514862 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:30.515522 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:30.515527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:30.515559 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:30.515752 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.515764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.517640 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.517646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.519878 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:30.519894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.521319 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.521327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:30.521527 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.521565 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 04:02:30.521670 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 04:02:30.521830 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 04:02:30.523234 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a3fc0f29cf4c8da0a763c6cadb4170" candidate_uuid: "e74115551f804886b526b17f5d35f717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5108a4a9d34a6899aecb645257d921" is_pre_election: true
I20260808 04:02:30.523993 26031 leader_election.cc:302] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e74115551f804886b526b17f5d35f717; no voters: 5e5108a4a9d34a6899aecb645257d921, a1f6341d7be7463f9c3d9fcfc5313fb5
I20260808 04:02:30.524554 26740 raft_consensus.cc:2726] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 04:02:30.544543 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.538282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.28.251.195:35513)
0808 04:02:30.538517 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:30.538526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:30.538900 (+ 374us) server_negotiation.cc:407] Connection header received
0808 04:02:30.539090 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:30.539094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:30.539199 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:30.539337 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:02:30.539886 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.540614 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.542303 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.542750 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.542855 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.543314 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:02:30.543988 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:02:30.544100 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:30.544212 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 04:02:30.544291 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:02:30.552968 26142 raft_consensus.cc:1261] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 1 FOLLOWER]: Refusing update from remote peer 5e5108a4a9d34a6899aecb645257d921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:30.552984 26383 raft_consensus.cc:1261] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 1 FOLLOWER]: Refusing update from remote peer 5e5108a4a9d34a6899aecb645257d921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:30.553867 26737 consensus_queue.cc:1035] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e74115551f804886b526b17f5d35f717" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 33743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:02:30.554262 26756 consensus_queue.cc:1035] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1f6341d7be7463f9c3d9fcfc5313fb5" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 43463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:02:30.568145 26767 mvcc.cc:205] Tried to move back new op lower bound from 7316118530254229504 to 7316118530103205888. Current Snapshot: MvccSnapshot[applied={T|T < 7316118530254229504}]
I20260808 04:02:30.583968 25901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:30.584254 25901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:30.588264 25901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16e44c0e4ea74882bd21dc51c3fbfa8c: Sending DeleteTablet for 3 replicas of tablet 23a3fc0f29cf4c8da0a763c6cadb4170
I20260808 04:02:30.589922 26573 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a3fc0f29cf4c8da0a763c6cadb4170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:30 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:58654
I20260808 04:02:30.589926 26106 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a3fc0f29cf4c8da0a763c6cadb4170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:30 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:52454
I20260808 04:02:30.590247 26348 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a3fc0f29cf4c8da0a763c6cadb4170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:30 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:48694
I20260808 04:02:30.594347 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 26009
I20260808 04:02:30.600445 26776 tablet_replica.cc:330] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: stopping tablet replica
I20260808 04:02:30.601089 26776 raft_consensus.cc:2229] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:02:30.602001 26776 raft_consensus.cc:2258] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:30.605530 26779 tablet_replica.cc:330] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: stopping tablet replica
I20260808 04:02:30.606091 26777 tablet_replica.cc:330] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: stopping tablet replica
I20260808 04:02:30.606086 26779 raft_consensus.cc:2229] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:30.606556 26777 raft_consensus.cc:2229] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:30.606709 26779 raft_consensus.cc:2258] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:30.607106 26777 raft_consensus.cc:2258] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:30.609304 26777 ts_tablet_manager.cc:1905] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:30.609320 26776 ts_tablet_manager.cc:1905] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 04:02:30 dist-test-slave-pww0 krb5kdc[25837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 04:02:30.613296 26779 ts_tablet_manager.cc:1905] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:30.617694 26777 ts_tablet_manager.cc:1918] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:30.617738 26776 ts_tablet_manager.cc:1918] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:30.617952 26776 log.cc:1196] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/23a3fc0f29cf4c8da0a763c6cadb4170
I20260808 04:02:30.618186 26777 log.cc:1196] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/23a3fc0f29cf4c8da0a763c6cadb4170
I20260808 04:02:30.618757 26776 ts_tablet_manager.cc:1939] T 23a3fc0f29cf4c8da0a763c6cadb4170 P 5e5108a4a9d34a6899aecb645257d921: Deleting consensus metadata
I20260808 04:02:30.619014 26777 ts_tablet_manager.cc:1939] T 23a3fc0f29cf4c8da0a763c6cadb4170 P e74115551f804886b526b17f5d35f717: Deleting consensus metadata
I20260808 04:02:30.621382 25882 catalog_manager.cc:4841] TS e74115551f804886b526b17f5d35f717 (127.28.251.193:33743): tablet 23a3fc0f29cf4c8da0a763c6cadb4170 (table test-table [id=d6c90182728f44d9bee4ab3be4fa41ba]) successfully deleted
I20260808 04:02:30.621963 25883 catalog_manager.cc:4841] TS 5e5108a4a9d34a6899aecb645257d921 (127.28.251.195:35513): tablet 23a3fc0f29cf4c8da0a763c6cadb4170 (table test-table [id=d6c90182728f44d9bee4ab3be4fa41ba]) successfully deleted
I20260808 04:02:30.625233 26779 ts_tablet_manager.cc:1918] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:30.625552 26779 log.cc:1196] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/23a3fc0f29cf4c8da0a763c6cadb4170
I20260808 04:02:30.626176 26779 ts_tablet_manager.cc:1939] T 23a3fc0f29cf4c8da0a763c6cadb4170 P a1f6341d7be7463f9c3d9fcfc5313fb5: Deleting consensus metadata
I20260808 04:02:30.627262 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.596056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.28.251.193:33743)
0808 04:02:30.596284 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:30.596290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:30.598616 (+ 2326us) server_negotiation.cc:407] Connection header received
0808 04:02:30.598724 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:30.598730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:30.598838 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:30.598978 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:30.602246 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.603530 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.605595 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.606357 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.606466 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.617944 (+ 11478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:30.617966 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:30.617989 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:30.618049 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:30.622558 (+ 4509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:30.623771 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:30.623782 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:30.623786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:30.624001 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:30.624577 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:30.624583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:30.624587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:30.625128 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:30.625464 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:30.626614 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0808 04:02:30.626880 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:02:30.628186 26216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:30.628364 25882 catalog_manager.cc:4841] TS a1f6341d7be7463f9c3d9fcfc5313fb5 (127.28.251.194:43463): tablet 23a3fc0f29cf4c8da0a763c6cadb4170 (table test-table [id=d6c90182728f44d9bee4ab3be4fa41ba]) successfully deleted
I20260808 04:02:30.723904 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 26009
I20260808 04:02:30.752478 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 26234
Aug 08 04:02:30 dist-test-slave-pww0 krb5kdc[25837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 04:02:30.769675 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.753429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.28.251.194:43463)
0808 04:02:30.753616 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:30.753623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:30.753705 (+ 82us) server_negotiation.cc:407] Connection header received
0808 04:02:30.753938 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:30.753942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:30.754022 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:30.754115 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:30.755565 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.756355 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.757821 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.758251 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.758375 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.763399 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:30.763422 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:30.763439 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:30.763520 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:30.766597 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:30.767612 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:30.767620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:30.767623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:30.767776 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:30.768203 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:30.768206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:30.768208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:30.768598 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:30.768856 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:30.769187 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 04:02:30.769381 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:02:30.770479 26448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:30.867573 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 26234
I20260808 04:02:30.905607 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 26464
Aug 08 04:02:30 dist-test-slave-pww0 krb5kdc[25837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 04:02:30.928611 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:30.906577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.28.251.195:35513)
0808 04:02:30.906849 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:30.906855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:30.906959 (+ 104us) server_negotiation.cc:407] Connection header received
0808 04:02:30.907175 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:30.907181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:30.907286 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:30.907391 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:30.909270 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.910544 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.912600 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:30.913099 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:30.913221 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:30.919398 (+ 6177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:30.919415 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:30.919436 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:30.919524 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:30.923336 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:30.924582 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:30.924593 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:30.924596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:30.924822 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:30.925509 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:30.925516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:30.925519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:30.926126 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:30.926507 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:30.927931 (+ 1424us) server_negotiation.cc:299] Negotiation successful
0808 04:02:30.928216 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 04:02:30.936479 26677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:31.078662 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 26464
I20260808 04:02:31.108536 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 25858
I20260808 04:02:31.109890 25952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:31.228303 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 25858
2026-08-08T04:02:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5394 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T04:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:02:31Z Disabled control of system clock
I20260808 04:02:31.340826 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:34221
--webserver_interface=127.28.251.254
--webserver_port=0
--builtin_ntp_servers=127.28.251.212:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:34221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:02:31.714509 26793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:31.714864 26793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:31.714951 26793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:31.722548 26793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:02:31.722690 26793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:31.722729 26793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:31.722796 26793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:02:31.722841 26793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:02:31.732544 26793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:34221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:34221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:31.733946 26793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:31.735877 26793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:31.748032 26802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:31.748081 26803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:31.748062 26805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:31.749985 26793 server_base.cc:1034] running on GCE node
I20260808 04:02:31.750932 26793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:31.752539 26793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:31.753778 26793 hybrid_clock.cc:648] HybridClock initialized: now 1786161751753710 us; error 54 us; skew 500 ppm
I20260808 04:02:31.754204 26793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:31.756814 26793 webserver.cc:460] Webserver started at http://127.28.251.254:34141/ using document root <none> and password file <none>
I20260808 04:02:31.757589 26793 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:31.757721 26793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:31.758060 26793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:31.761284 26793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "651b5bcc9a174a138bf6312ebfd63f16"
format_stamp: "Formatted at 2026-08-08 04:02:31 on dist-test-slave-pww0"
server_key: "ce8ef2a617e504c7cea3d4f6018c3311"
server_key_iv: "a40fc93804e07bdec447723bd7a6225a"
server_key_version: "encryptionkey@0"
I20260808 04:02:31.762164 26793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "651b5bcc9a174a138bf6312ebfd63f16"
format_stamp: "Formatted at 2026-08-08 04:02:31 on dist-test-slave-pww0"
server_key: "ce8ef2a617e504c7cea3d4f6018c3311"
server_key_iv: "a40fc93804e07bdec447723bd7a6225a"
server_key_version: "encryptionkey@0"
I20260808 04:02:31.767926 26793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 04:02:31.771674 26812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:31.773146 26793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:02:31.773345 26793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "651b5bcc9a174a138bf6312ebfd63f16"
format_stamp: "Formatted at 2026-08-08 04:02:31 on dist-test-slave-pww0"
server_key: "ce8ef2a617e504c7cea3d4f6018c3311"
server_key_iv: "a40fc93804e07bdec447723bd7a6225a"
server_key_version: "encryptionkey@0"
I20260808 04:02:31.773586 26793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:31.786494 26793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:31.787429 26793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:31.787822 26793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:31.806577 26793 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:34221
I20260808 04:02:31.806597 26895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:34221 every 8 connection(s)
I20260808 04:02:31.808048 26793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:02:31.808995 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 26793
I20260808 04:02:31.809216 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:02:31.809747 29679 external_mini_cluster.cc:1428] Setting key e4a4d88c3dcf2eede489fedc2ba6193b
I20260808 04:02:31.812844 26897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:31.825837 26897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16: Bootstrap starting.
I20260808 04:02:31.826453 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:31.811949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.28.251.254:34221)
0808 04:02:31.813186 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:31.813199 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:31.813254 (+ 55us) server_negotiation.cc:407] Connection header received
0808 04:02:31.815926 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:31.815961 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:31.816755 (+ 794us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:31.817272 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:31.817945 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:31.819502 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:31.821202 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:31.821999 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:31.822218 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:31.823694 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:31.823704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:31.823722 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:31.824414 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:31.824520 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:31.824547 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:31.824682 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:31.824960 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 04:02:31.825326 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":409,"threads_started":1}
I20260808 04:02:31.829779 26897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:31.830938 26897 log.cc:824] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:31.833953 26897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16: No bootstrap required, opened a new log
I20260808 04:02:31.842473 26897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651b5bcc9a174a138bf6312ebfd63f16" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34221 } }
I20260808 04:02:31.842895 26897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:31.842984 26897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651b5bcc9a174a138bf6312ebfd63f16, State: Initialized, Role: FOLLOWER
I20260808 04:02:31.843608 26897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651b5bcc9a174a138bf6312ebfd63f16" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34221 } }
I20260808 04:02:31.843840 26897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:02:31.843950 26897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:02:31.844091 26897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:31.846604 26897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651b5bcc9a174a138bf6312ebfd63f16" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34221 } }
I20260808 04:02:31.847237 26897 leader_election.cc:302] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 651b5bcc9a174a138bf6312ebfd63f16; no voters:
I20260808 04:02:31.847944 26897 leader_election.cc:288] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:02:31.848096 26909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:31.848894 26909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [term 1 LEADER]: Becoming Leader. State: Replica: 651b5bcc9a174a138bf6312ebfd63f16, State: Running, Role: LEADER
I20260808 04:02:31.849476 26909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651b5bcc9a174a138bf6312ebfd63f16" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34221 } }
I20260808 04:02:31.850168 26897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:02:31.852720 26915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 651b5bcc9a174a138bf6312ebfd63f16. Latest consensus state: current_term: 1 leader_uuid: "651b5bcc9a174a138bf6312ebfd63f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651b5bcc9a174a138bf6312ebfd63f16" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34221 } } }
I20260808 04:02:31.853371 26915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:31.853375 26910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "651b5bcc9a174a138bf6312ebfd63f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651b5bcc9a174a138bf6312ebfd63f16" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34221 } } }
I20260808 04:02:31.853716 26910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:31.855724 26927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:02:31.861027 26927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:02:31.870990 26927 catalog_manager.cc:1347] Generated new cluster ID: 84cfe179de1f457fbdf0a4de84584855
I20260808 04:02:31.871168 26927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:02:31.884788 26927 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:02:31.886205 26927 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:02:31.892743 26927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16: Generated new TSK 0
I20260808 04:02:31.893927 26927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:02:31.898587 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:34221
--builtin_ntp_servers=127.28.251.212:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:02:32.279210 26947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:32.279589 26947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:32.279754 26947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:32.287536 26947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:32.287679 26947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:32.287853 26947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:02:32.297679 26947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:34221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:32.299099 26947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:32.301102 26947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:32.314954 26954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:32.315068 26957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:32.315088 26955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:32.315979 26947 server_base.cc:1034] running on GCE node
I20260808 04:02:32.316705 26947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:32.317896 26947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:32.319118 26947 hybrid_clock.cc:648] HybridClock initialized: now 1786161752319050 us; error 68 us; skew 500 ppm
I20260808 04:02:32.319509 26947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:32.322101 26947 webserver.cc:460] Webserver started at http://127.28.251.193:38227/ using document root <none> and password file <none>
I20260808 04:02:32.322880 26947 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:32.322997 26947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:32.323333 26947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:32.326570 26947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "4608943539f9463c94112e0f256816d8"
format_stamp: "Formatted at 2026-08-08 04:02:32 on dist-test-slave-pww0"
server_key: "4a01923c1eda299746c6875ca7a5c5e3"
server_key_iv: "4ac6e1c60e44c37d45397fa97b5899b3"
server_key_version: "encryptionkey@0"
I20260808 04:02:32.327441 26947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "4608943539f9463c94112e0f256816d8"
format_stamp: "Formatted at 2026-08-08 04:02:32 on dist-test-slave-pww0"
server_key: "4a01923c1eda299746c6875ca7a5c5e3"
server_key_iv: "4ac6e1c60e44c37d45397fa97b5899b3"
server_key_version: "encryptionkey@0"
I20260808 04:02:32.334124 26947 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 04:02:32.338439 26965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:32.340065 26947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 04:02:32.340272 26947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "4608943539f9463c94112e0f256816d8"
format_stamp: "Formatted at 2026-08-08 04:02:32 on dist-test-slave-pww0"
server_key: "4a01923c1eda299746c6875ca7a5c5e3"
server_key_iv: "4ac6e1c60e44c37d45397fa97b5899b3"
server_key_version: "encryptionkey@0"
I20260808 04:02:32.340507 26947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:32.355434 26947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:32.356488 26947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:32.356818 26947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:32.357889 26947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:32.359561 26947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:32.359669 26947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:32.359799 26947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:32.359875 26947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:32.393060 26947 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:34055
I20260808 04:02:32.393149 27152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:34055 every 8 connection(s)
I20260808 04:02:32.394439 26947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:02:32.396780 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 26947
I20260808 04:02:32.396977 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 04:02:32.397708 29679 external_mini_cluster.cc:1428] Setting key 602bb81634f003bd6cecad768d8fefc9
I20260808 04:02:32.400897 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:34221
--builtin_ntp_servers=127.28.251.212:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:02:32.408272 27158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:32.397417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:58839 (local address 127.28.251.254:34221)
0808 04:02:32.397885 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:32.397890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:32.399449 (+ 1559us) server_negotiation.cc:407] Connection header received
0808 04:02:32.401488 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:32.401494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:32.401574 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:32.401688 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:32.402798 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:32.403801 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:32.405417 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:32.405944 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:32.406072 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:32.407398 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:32.407403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:32.407407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:32.407432 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:32.407504 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:32.407521 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:32.407608 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:32.407814 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 04:02:32.407916 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260808 04:02:32.409303 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:32.398102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:34221 (local address 127.28.251.193:58839)
0808 04:02:32.399261 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:32.399325 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:32.401198 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:32.401907 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:32.401916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:32.402011 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:32.402613 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:32.402631 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:32.403961 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:32.403965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:32.405263 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:32.405276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:32.406150 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:32.406155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:32.406372 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:32.407065 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:02:32.407067 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:32.407111 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:02:32.407195 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:32.407608 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:32.407616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:32.407623 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:02:32.407776 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 04:02:32.408135 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":333,"threads_started":1}
I20260808 04:02:32.411008 27156 heartbeater.cc:347] Connected to a master server at 127.28.251.254:34221
I20260808 04:02:32.411319 27156 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:32.412209 27156 heartbeater.cc:511] Master 127.28.251.254:34221 requested a full tablet report, sending...
I20260808 04:02:32.414453 26843 ts_manager.cc:194] Registered new tserver with Master: 4608943539f9463c94112e0f256816d8 (127.28.251.193:34055)
I20260808 04:02:32.416041 26843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.193:58839
W20260808 04:02:32.769958 27160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:32.770265 27160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:32.770381 27160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:32.778230 27160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:32.778362 27160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:32.778477 27160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:02:32.788112 27160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:34221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:32.789502 27160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:32.791496 27160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:32.805922 27168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:32.806421 27169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:32.805950 27174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:32.809020 27160 server_base.cc:1034] running on GCE node
I20260808 04:02:32.810081 27160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:32.811355 27160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:32.812674 27160 hybrid_clock.cc:648] HybridClock initialized: now 1786161752812609 us; error 56 us; skew 500 ppm
I20260808 04:02:32.813066 27160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:32.816004 27160 webserver.cc:460] Webserver started at http://127.28.251.194:38353/ using document root <none> and password file <none>
I20260808 04:02:32.816895 27160 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:32.817057 27160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:32.817440 27160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:32.821779 27160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "70ed028a39cc44a293ceb10fae960e51"
format_stamp: "Formatted at 2026-08-08 04:02:32 on dist-test-slave-pww0"
server_key: "d5f36978dfa81bac1e5c4424a245cd4c"
server_key_iv: "1663c8e8b497ff49b31450cbc09b2a82"
server_key_version: "encryptionkey@0"
I20260808 04:02:32.822912 27160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "70ed028a39cc44a293ceb10fae960e51"
format_stamp: "Formatted at 2026-08-08 04:02:32 on dist-test-slave-pww0"
server_key: "d5f36978dfa81bac1e5c4424a245cd4c"
server_key_iv: "1663c8e8b497ff49b31450cbc09b2a82"
server_key_version: "encryptionkey@0"
I20260808 04:02:32.829579 27160 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260808 04:02:32.833580 27190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:32.835049 27160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:02:32.835201 27160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "70ed028a39cc44a293ceb10fae960e51"
format_stamp: "Formatted at 2026-08-08 04:02:32 on dist-test-slave-pww0"
server_key: "d5f36978dfa81bac1e5c4424a245cd4c"
server_key_iv: "1663c8e8b497ff49b31450cbc09b2a82"
server_key_version: "encryptionkey@0"
I20260808 04:02:32.835400 27160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:32.860167 27160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:32.861109 27160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:32.861408 27160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:32.862515 27160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:32.864171 27160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:32.864267 27160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:32.864356 27160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:32.864413 27160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:32.896842 27160 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:41197
I20260808 04:02:32.896925 27382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:41197 every 8 connection(s)
I20260808 04:02:32.898248 27160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:02:32.901769 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 27160
I20260808 04:02:32.901968 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 04:02:32.902567 29679 external_mini_cluster.cc:1428] Setting key ffd94352f582318634766e0e886fe766
I20260808 04:02:32.909682 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:34221
--builtin_ntp_servers=127.28.251.212:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:02:32.912140 27158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:32.901037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:34059 (local address 127.28.251.254:34221)
0808 04:02:32.901218 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:32.901223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:32.902921 (+ 1698us) server_negotiation.cc:407] Connection header received
0808 04:02:32.904887 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:32.904893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:32.904981 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:32.905075 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:32.906277 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:32.907048 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:32.909078 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:32.909485 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:32.909581 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:32.911346 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:32.911351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:32.911354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:32.911375 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:32.911408 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:32.911421 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:32.911559 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:32.911812 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 04:02:32.911898 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:02:32.913643 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:32.901644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:34221 (local address 127.28.251.194:34059)
0808 04:02:32.902733 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:32.902793 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:32.904578 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:32.905341 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:32.905350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:32.905454 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:32.906071 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:32.906089 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:32.907231 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:32.907236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:32.908905 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:32.908922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:32.909763 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:32.909773 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:32.910112 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:32.910944 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:02:32.910947 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:32.911003 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:02:32.911120 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:32.911563 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:32.911576 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:32.911585 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:02:32.911811 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 04:02:32.912240 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":379,"threads_started":1}
I20260808 04:02:32.915272 27383 heartbeater.cc:347] Connected to a master server at 127.28.251.254:34221
I20260808 04:02:32.915669 27383 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:32.916472 27383 heartbeater.cc:511] Master 127.28.251.254:34221 requested a full tablet report, sending...
I20260808 04:02:32.918067 26843 ts_manager.cc:194] Registered new tserver with Master: 70ed028a39cc44a293ceb10fae960e51 (127.28.251.194:41197)
I20260808 04:02:32.918962 26843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.194:34059
W20260808 04:02:33.297159 27391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:33.297485 27391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:33.297623 27391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:33.305322 27391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:33.305469 27391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:33.305625 27391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:02:33.315810 27391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:34221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:33.317252 27391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:33.319260 27391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:33.333158 27399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:33.333331 27405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:33.333211 27402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:33.334136 27391 server_base.cc:1034] running on GCE node
I20260808 04:02:33.334815 27391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:33.335937 27391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:33.337225 27391 hybrid_clock.cc:648] HybridClock initialized: now 1786161753337165 us; error 49 us; skew 500 ppm
I20260808 04:02:33.337606 27391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:33.340301 27391 webserver.cc:460] Webserver started at http://127.28.251.195:46241/ using document root <none> and password file <none>
I20260808 04:02:33.341091 27391 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:33.341212 27391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:33.341549 27391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:33.344796 27391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "b48b41ec43a24816bd0bc1468caed0a6"
format_stamp: "Formatted at 2026-08-08 04:02:33 on dist-test-slave-pww0"
server_key: "891c4850d7ab107d38fbe326fbf57dd6"
server_key_iv: "ff7adf0a1eb513f8d3f34cb06f2c7768"
server_key_version: "encryptionkey@0"
I20260808 04:02:33.345809 27391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "b48b41ec43a24816bd0bc1468caed0a6"
format_stamp: "Formatted at 2026-08-08 04:02:33 on dist-test-slave-pww0"
server_key: "891c4850d7ab107d38fbe326fbf57dd6"
server_key_iv: "ff7adf0a1eb513f8d3f34cb06f2c7768"
server_key_version: "encryptionkey@0"
I20260808 04:02:33.351776 27391 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260808 04:02:33.355959 27415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:33.357597 27391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:02:33.357769 27391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "b48b41ec43a24816bd0bc1468caed0a6"
format_stamp: "Formatted at 2026-08-08 04:02:33 on dist-test-slave-pww0"
server_key: "891c4850d7ab107d38fbe326fbf57dd6"
server_key_iv: "ff7adf0a1eb513f8d3f34cb06f2c7768"
server_key_version: "encryptionkey@0"
I20260808 04:02:33.357980 27391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:33.389752 27391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:33.390702 27391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:33.390988 27391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:33.392110 27391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:33.393823 27391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:33.393927 27391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:33.394052 27391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:33.394124 27391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:33.419230 27156 heartbeater.cc:503] Master 127.28.251.254:34221 was elected leader, sending a full tablet report...
I20260808 04:02:33.426397 27391 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:37241
I20260808 04:02:33.426471 27603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:37241 every 8 connection(s)
I20260808 04:02:33.427878 27391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:02:33.428714 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 27391
I20260808 04:02:33.428923 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 04:02:33.429464 29679 external_mini_cluster.cc:1428] Setting key a336627afd813a5712d1c90cd1df57fc
I20260808 04:02:33.441617 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.430714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:47495 (local address 127.28.251.254:34221)
0808 04:02:33.431328 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:33.431333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:33.432766 (+ 1433us) server_negotiation.cc:407] Connection header received
0808 04:02:33.434642 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:33.434648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:33.434731 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:33.434827 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:33.436113 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.436946 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.438612 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.439049 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.439164 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.440657 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:33.440662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:33.440665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:33.440687 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:33.440729 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:33.440742 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:33.440819 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:33.441091 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 04:02:33.441196 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":381,"threads_started":1}
I20260808 04:02:33.442683 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.431565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:34221 (local address 127.28.251.195:47495)
0808 04:02:33.432611 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:33.432662 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:33.434359 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:33.435112 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:33.435121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:33.435216 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:33.435932 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.435953 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.437128 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.437133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.438450 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.438464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.439318 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.439327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.439619 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.440305 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:02:33.440308 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:33.440351 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:02:33.440442 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:33.440861 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:33.440872 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:33.440879 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:02:33.441039 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 04:02:33.441483 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":343,"threads_started":1}
I20260808 04:02:33.444271 27604 heartbeater.cc:347] Connected to a master server at 127.28.251.254:34221
I20260808 04:02:33.444585 27604 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:33.445582 27604 heartbeater.cc:511] Master 127.28.251.254:34221 requested a full tablet report, sending...
I20260808 04:02:33.447006 26842 ts_manager.cc:194] Registered new tserver with Master: b48b41ec43a24816bd0bc1468caed0a6 (127.28.251.195:37241)
I20260808 04:02:33.447981 26842 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.195:47495
I20260808 04:02:33.458524 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:02:33.469471 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.463172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.28.251.254:34221)
0808 04:02:33.463357 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:33.463362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:33.463800 (+ 438us) server_negotiation.cc:407] Connection header received
0808 04:02:33.463946 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:33.463952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:33.464033 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:33.464146 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:33.464689 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.465429 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.467003 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.467503 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.467627 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.468841 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:33.468846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:33.468849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:33.468874 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:33.468914 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:33.468927 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:33.469005 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:33.469110 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 04:02:33.469202 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:02:33.474283 26842 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:02:33.482046 26842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:02:33.512835 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.495516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.28.251.193:34055)
0808 04:02:33.497805 (+ 2289us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:33.497816 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:33.497853 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:02:33.497967 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:33.497976 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:33.499686 (+ 1710us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:33.500823 (+ 1137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:33.501140 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.504356 (+ 3216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.509212 (+ 4856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.510856 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.511885 (+ 1029us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.512133 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:33.512214 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 04:02:33.512426 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2058,"thread_start_us":1157,"threads_started":1}
I20260808 04:02:33.513469 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.495224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:34055 (local address 127.0.0.1:60482)
0808 04:02:33.496131 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:33.496180 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:33.496388 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:33.500010 (+ 3622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:33.500019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:33.500068 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:33.500551 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.500564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.504545 (+ 3981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.504553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.508856 (+ 4303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.508883 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.511098 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.511107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.511360 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.511446 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 04:02:33.511587 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 04:02:33.511808 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":441,"threads_started":1}
I20260808 04:02:33.518251 27632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.496642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:37241 (local address 127.0.0.1:56946)
0808 04:02:33.497210 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:33.497230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:33.497354 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:33.510437 (+ 13083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:33.510442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:33.510470 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:33.510844 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.510857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.513510 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.513516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.515557 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.515574 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.517439 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.517446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.517619 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.517655 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 04:02:33.517769 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 04:02:33.517917 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":26,"thread_start_us":984,"threads_started":1}
I20260808 04:02:33.518347 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.497308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.28.251.195:37241)
0808 04:02:33.509641 (+ 12333us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:33.509661 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:33.509705 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:02:33.509827 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:33.509836 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:33.510156 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:33.511052 (+ 896us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:33.511379 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.513325 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.515771 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.517199 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.517423 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.517629 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:33.517836 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 04:02:33.517977 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12163,"thread_start_us":13029,"threads_started":1}
I20260808 04:02:33.517818 27035 tablet_service.cc:1467] Processing CreateTablet for tablet 933e82e027614ebbb2c57174a359e1e9 (DEFAULT_TABLE table=test-table [id=9dae0388828941c9973d5208e66e7977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:33.519322 27636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.496984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41197 (local address 127.0.0.1:56866)
0808 04:02:33.510075 (+ 13091us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:33.510096 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:33.510235 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:33.510931 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:33.510936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:33.510961 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:33.511402 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.511416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.513419 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.513425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.515443 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.515493 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.518533 (+ 3040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.518540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.518728 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.518763 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:02:33.518838 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 04:02:33.518970 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12963,"mutex_wait_us":37,"thread_start_us":12817,"threads_started":1}
I20260808 04:02:33.520089 27035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933e82e027614ebbb2c57174a359e1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:33.520207 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.497255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.28.251.194:41197)
0808 04:02:33.509603 (+ 12348us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:33.509614 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:33.510323 (+ 709us) server_negotiation.cc:407] Connection header received
0808 04:02:33.510456 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:33.510464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:33.510707 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:33.510939 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:33.511656 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.513250 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.515696 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.517199 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.517386 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.517630 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:33.519701 (+ 2071us) server_negotiation.cc:299] Negotiation successful
0808 04:02:33.519894 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12133,"thread_start_us":12768,"threads_started":1}
I20260808 04:02:33.522774 27504 tablet_service.cc:1467] Processing CreateTablet for tablet 933e82e027614ebbb2c57174a359e1e9 (DEFAULT_TABLE table=test-table [id=9dae0388828941c9973d5208e66e7977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:33.524142 27504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933e82e027614ebbb2c57174a359e1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:33.524117 27268 tablet_service.cc:1467] Processing CreateTablet for tablet 933e82e027614ebbb2c57174a359e1e9 (DEFAULT_TABLE table=test-table [id=9dae0388828941c9973d5208e66e7977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:33.525601 27268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933e82e027614ebbb2c57174a359e1e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:33.537556 27644 tablet_bootstrap.cc:492] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: Bootstrap starting.
I20260808 04:02:33.539779 27646 tablet_bootstrap.cc:492] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: Bootstrap starting.
I20260808 04:02:33.541808 27644 tablet_bootstrap.cc:654] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:33.542965 27644 log.cc:824] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:33.544044 27646 tablet_bootstrap.cc:654] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:33.544493 27647 tablet_bootstrap.cc:492] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: Bootstrap starting.
I20260808 04:02:33.545776 27646 log.cc:824] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:33.548405 27644 tablet_bootstrap.cc:492] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: No bootstrap required, opened a new log
I20260808 04:02:33.548733 27644 ts_tablet_manager.cc:1397] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260808 04:02:33.548974 27647 tablet_bootstrap.cc:654] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:33.550009 27647 log.cc:824] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:33.550562 27646 tablet_bootstrap.cc:492] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: No bootstrap required, opened a new log
I20260808 04:02:33.550905 27646 ts_tablet_manager.cc:1397] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20260808 04:02:33.552835 27647 tablet_bootstrap.cc:492] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: No bootstrap required, opened a new log
I20260808 04:02:33.553098 27647 ts_tablet_manager.cc:1397] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 04:02:33.559834 27644 raft_consensus.cc:348] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608943539f9463c94112e0f256816d8" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 34055 } } peers { permanent_uuid: "b48b41ec43a24816bd0bc1468caed0a6" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37241 } } peers { permanent_uuid: "70ed028a39cc44a293ceb10fae960e51" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41197 } }
I20260808 04:02:33.560474 27644 raft_consensus.cc:374] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:33.560608 27644 raft_consensus.cc:729] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4608943539f9463c94112e0f256816d8, State: Initialized, Role: FOLLOWER
I20260808 04:02:33.561378 27644 consensus_queue.cc:260] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608943539f9463c94112e0f256816d8" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 34055 } } peers { permanent_uuid: "b48b41ec43a24816bd0bc1468caed0a6" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37241 } } peers { permanent_uuid: "70ed028a39cc44a293ceb10fae960e51" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41197 } }
I20260808 04:02:33.561323 27647 raft_consensus.cc:348] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608943539f9463c94112e0f256816d8" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 34055 } } peers { permanent_uuid: "b48b41ec43a24816bd0bc1468caed0a6" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37241 } } peers { permanent_uuid: "70ed028a39cc44a293ceb10fae960e51" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41197 } }
I20260808 04:02:33.561941 27647 raft_consensus.cc:374] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:33.562093 27647 raft_consensus.cc:729] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ed028a39cc44a293ceb10fae960e51, State: Initialized, Role: FOLLOWER
I20260808 04:02:33.561873 27646 raft_consensus.cc:348] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608943539f9463c94112e0f256816d8" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 34055 } } peers { permanent_uuid: "b48b41ec43a24816bd0bc1468caed0a6" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37241 } } peers { permanent_uuid: "70ed028a39cc44a293ceb10fae960e51" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41197 } }
I20260808 04:02:33.562348 27646 raft_consensus.cc:374] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:33.562458 27646 raft_consensus.cc:729] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48b41ec43a24816bd0bc1468caed0a6, State: Initialized, Role: FOLLOWER
I20260808 04:02:33.563054 27646 consensus_queue.cc:260] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608943539f9463c94112e0f256816d8" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 34055 } } peers { permanent_uuid: "b48b41ec43a24816bd0bc1468caed0a6" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37241 } } peers { permanent_uuid: "70ed028a39cc44a293ceb10fae960e51" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41197 } }
I20260808 04:02:33.563657 27644 ts_tablet_manager.cc:1428] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260808 04:02:33.563849 27647 consensus_queue.cc:260] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608943539f9463c94112e0f256816d8" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 34055 } } peers { permanent_uuid: "b48b41ec43a24816bd0bc1468caed0a6" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37241 } } peers { permanent_uuid: "70ed028a39cc44a293ceb10fae960e51" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41197 } }
I20260808 04:02:33.564738 27604 heartbeater.cc:503] Master 127.28.251.254:34221 was elected leader, sending a full tablet report...
I20260808 04:02:33.565240 27383 heartbeater.cc:503] Master 127.28.251.254:34221 was elected leader, sending a full tablet report...
I20260808 04:02:33.565441 27646 ts_tablet_manager.cc:1428] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260808 04:02:33.566869 27647 ts_tablet_manager.cc:1428] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: Time spent starting tablet: real 0.014s user 0.004s sys 0.007s
I20260808 04:02:33.594511 27659 raft_consensus.cc:482] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:33.594874 27659 raft_consensus.cc:504] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608943539f9463c94112e0f256816d8" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 34055 } } peers { permanent_uuid: "b48b41ec43a24816bd0bc1468caed0a6" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37241 } } peers { permanent_uuid: "70ed028a39cc44a293ceb10fae960e51" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41197 } }
I20260808 04:02:33.596750 27659 leader_election.cc:288] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4608943539f9463c94112e0f256816d8 (127.28.251.193:34055), 70ed028a39cc44a293ceb10fae960e51 (127.28.251.194:41197)
I20260808 04:02:33.602761 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.597555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:47105 (local address 127.28.251.194:41197)
0808 04:02:33.597783 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:33.597788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:33.597803 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:02:33.597873 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:33.597877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:33.597946 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:33.598025 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:33.598529 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.599233 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.600985 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.601858 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.602007 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.602085 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:33.602348 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 04:02:33.602487 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 04:02:33.602761 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.597047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41197 (local address 127.28.251.195:47105)
0808 04:02:33.597367 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:33.597387 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:33.597513 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:33.598138 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:33.598143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:33.598179 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:33.598362 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.598372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.599379 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.599383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.600810 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.600826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.602013 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.602019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.602192 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.602217 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:02:33.602302 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:02:33.602461 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260808 04:02:33.603809 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.597484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:54739 (local address 127.28.251.193:34055)
0808 04:02:33.597706 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:33.597711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:33.598544 (+ 833us) server_negotiation.cc:407] Connection header received
0808 04:02:33.598708 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:33.598712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:33.598783 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:33.598860 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:33.599418 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.600221 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.602115 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.602992 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.603158 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.603229 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:33.603517 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 04:02:33.603625 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:02:33.603940 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.597203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:34055 (local address 127.28.251.195:54739)
0808 04:02:33.598425 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:33.598439 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:33.598566 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:33.598907 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:33.598911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:33.598935 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:33.599244 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.599253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.600383 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.600387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.601920 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:33.601934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.603191 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.603197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:33.603336 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.603361 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:02:33.603443 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 04:02:33.603600 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":428,"thread_start_us":401,"threads_started":1}
I20260808 04:02:33.604067 27307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933e82e027614ebbb2c57174a359e1e9" candidate_uuid: "b48b41ec43a24816bd0bc1468caed0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ed028a39cc44a293ceb10fae960e51" is_pre_election: true
I20260808 04:02:33.604701 27307 raft_consensus.cc:2454] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b48b41ec43a24816bd0bc1468caed0a6 in term 0.
I20260808 04:02:33.604990 27074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933e82e027614ebbb2c57174a359e1e9" candidate_uuid: "b48b41ec43a24816bd0bc1468caed0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4608943539f9463c94112e0f256816d8" is_pre_election: true
I20260808 04:02:33.605491 27420 leader_election.cc:302] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ed028a39cc44a293ceb10fae960e51, b48b41ec43a24816bd0bc1468caed0a6; no voters:
I20260808 04:02:33.605571 27074 raft_consensus.cc:2454] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b48b41ec43a24816bd0bc1468caed0a6 in term 0.
I20260808 04:02:33.605911 27659 raft_consensus.cc:2781] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:02:33.606092 27659 raft_consensus.cc:482] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:02:33.606204 27659 raft_consensus.cc:3037] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:33.608584 27659 raft_consensus.cc:504] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608943539f9463c94112e0f256816d8" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 34055 } } peers { permanent_uuid: "b48b41ec43a24816bd0bc1468caed0a6" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37241 } } peers { permanent_uuid: "70ed028a39cc44a293ceb10fae960e51" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41197 } }
I20260808 04:02:33.609207 27659 leader_election.cc:288] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4608943539f9463c94112e0f256816d8 (127.28.251.193:34055), 70ed028a39cc44a293ceb10fae960e51 (127.28.251.194:41197)
I20260808 04:02:33.609772 27307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933e82e027614ebbb2c57174a359e1e9" candidate_uuid: "b48b41ec43a24816bd0bc1468caed0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ed028a39cc44a293ceb10fae960e51"
I20260808 04:02:33.609795 27074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933e82e027614ebbb2c57174a359e1e9" candidate_uuid: "b48b41ec43a24816bd0bc1468caed0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4608943539f9463c94112e0f256816d8"
I20260808 04:02:33.609983 27307 raft_consensus.cc:3037] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:33.610009 27074 raft_consensus.cc:3037] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:33.612385 27074 raft_consensus.cc:2454] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b48b41ec43a24816bd0bc1468caed0a6 in term 1.
I20260808 04:02:33.612401 27307 raft_consensus.cc:2454] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b48b41ec43a24816bd0bc1468caed0a6 in term 1.
I20260808 04:02:33.612864 27420 leader_election.cc:302] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ed028a39cc44a293ceb10fae960e51, b48b41ec43a24816bd0bc1468caed0a6; no voters:
I20260808 04:02:33.613210 27659 raft_consensus.cc:2781] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:33.613687 27659 raft_consensus.cc:686] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 1 LEADER]: Becoming Leader. State: Replica: b48b41ec43a24816bd0bc1468caed0a6, State: Running, Role: LEADER
I20260808 04:02:33.614189 27659 consensus_queue.cc:237] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608943539f9463c94112e0f256816d8" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 34055 } } peers { permanent_uuid: "b48b41ec43a24816bd0bc1468caed0a6" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37241 } } peers { permanent_uuid: "70ed028a39cc44a293ceb10fae960e51" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41197 } }
I20260808 04:02:33.619115 26842 catalog_manager.cc:5495] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b48b41ec43a24816bd0bc1468caed0a6 (127.28.251.195). New cstate: current_term: 1 leader_uuid: "b48b41ec43a24816bd0bc1468caed0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4608943539f9463c94112e0f256816d8" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b48b41ec43a24816bd0bc1468caed0a6" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 37241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70ed028a39cc44a293ceb10fae960e51" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41197 } health_report { overall_health: UNKNOWN } } }
W20260808 04:02:33.647053 27157 tablet.cc:2369] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:02:33.650568 27387 tablet.cc:2369] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:02:33.664654 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.659068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.28.251.195:37241)
0808 04:02:33.659250 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:33.659255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:33.659340 (+ 85us) server_negotiation.cc:407] Connection header received
0808 04:02:33.659569 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:33.659573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:33.659660 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:33.659802 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:02:33.660164 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.660962 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.662495 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.662941 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.663039 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.663555 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:02:33.664094 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:02:33.664206 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:33.664319 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 04:02:33.664393 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:02:33.673530 27307 raft_consensus.cc:1261] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51 [term 1 FOLLOWER]: Refusing update from remote peer b48b41ec43a24816bd0bc1468caed0a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:33.673530 27074 raft_consensus.cc:1261] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8 [term 1 FOLLOWER]: Refusing update from remote peer b48b41ec43a24816bd0bc1468caed0a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:33.674711 27659 consensus_queue.cc:1035] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4608943539f9463c94112e0f256816d8" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:02:33.675068 27663 consensus_queue.cc:1035] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ed028a39cc44a293ceb10fae960e51" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:02:33.679725 27605 tablet.cc:2369] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:02:33.709523 26842 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:33.709771 26842 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:33.713587 26842 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 651b5bcc9a174a138bf6312ebfd63f16: Sending DeleteTablet for 3 replicas of tablet 933e82e027614ebbb2c57174a359e1e9
I20260808 04:02:33.714959 27035 tablet_service.cc:1514] Processing DeleteTablet for tablet 933e82e027614ebbb2c57174a359e1e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:33 UTC) from {username='slave'} at 127.0.0.1:60482
I20260808 04:02:33.715198 27504 tablet_service.cc:1514] Processing DeleteTablet for tablet 933e82e027614ebbb2c57174a359e1e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:33 UTC) from {username='slave'} at 127.0.0.1:56946
I20260808 04:02:33.715854 27268 tablet_service.cc:1514] Processing DeleteTablet for tablet 933e82e027614ebbb2c57174a359e1e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:33 UTC) from {username='slave'} at 127.0.0.1:56866
I20260808 04:02:33.716032 27690 tablet_replica.cc:330] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: stopping tablet replica
I20260808 04:02:33.716655 27690 raft_consensus.cc:2229] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:33.717361 27690 raft_consensus.cc:2258] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:33.717892 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 26947
I20260808 04:02:33.728695 27690 ts_tablet_manager.cc:1905] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:33.735682 27692 tablet_replica.cc:330] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: stopping tablet replica
I20260808 04:02:33.736287 27692 raft_consensus.cc:2229] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:33.736953 27692 raft_consensus.cc:2258] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:33.742260 27692 ts_tablet_manager.cc:1905] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:33.747203 27690 ts_tablet_manager.cc:1918] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:33.747400 27690 log.cc:1196] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/933e82e027614ebbb2c57174a359e1e9
I20260808 04:02:33.747879 27690 ts_tablet_manager.cc:1939] T 933e82e027614ebbb2c57174a359e1e9 P 4608943539f9463c94112e0f256816d8: Deleting consensus metadata
I20260808 04:02:33.749660 26813 catalog_manager.cc:4841] TS 4608943539f9463c94112e0f256816d8 (127.28.251.193:34055): tablet 933e82e027614ebbb2c57174a359e1e9 (table test-table [id=9dae0388828941c9973d5208e66e7977]) successfully deleted
I20260808 04:02:33.751317 27691 tablet_replica.cc:330] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: stopping tablet replica
I20260808 04:02:33.752013 27691 raft_consensus.cc:2229] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:02:33.752496 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.719759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.28.251.193:34055)
0808 04:02:33.719969 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:33.719976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:33.739921 (+ 19945us) server_negotiation.cc:407] Connection header received
0808 04:02:33.740119 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:33.740124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:33.740282 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:33.740395 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:33.741267 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.742311 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.744336 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.745044 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.745152 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.750417 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:33.750429 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:33.750449 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:33.751325 (+ 876us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:33.751403 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:33.751436 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:33.751560 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:33.751992 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 04:02:33.752138 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:02:33.752884 27691 raft_consensus.cc:2258] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:33.754012 27147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:33.755597 27691 ts_tablet_manager.cc:1905] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:33.757161 27692 ts_tablet_manager.cc:1918] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:33.757385 27692 log.cc:1196] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/933e82e027614ebbb2c57174a359e1e9
I20260808 04:02:33.758001 27692 ts_tablet_manager.cc:1939] T 933e82e027614ebbb2c57174a359e1e9 P 70ed028a39cc44a293ceb10fae960e51: Deleting consensus metadata
I20260808 04:02:33.760290 26818 catalog_manager.cc:4841] TS 70ed028a39cc44a293ceb10fae960e51 (127.28.251.194:41197): tablet 933e82e027614ebbb2c57174a359e1e9 (table test-table [id=9dae0388828941c9973d5208e66e7977]) successfully deleted
I20260808 04:02:33.763101 27691 ts_tablet_manager.cc:1918] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:33.763289 27691 log.cc:1196] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/933e82e027614ebbb2c57174a359e1e9
I20260808 04:02:33.763901 27691 ts_tablet_manager.cc:1939] T 933e82e027614ebbb2c57174a359e1e9 P b48b41ec43a24816bd0bc1468caed0a6: Deleting consensus metadata
I20260808 04:02:33.765630 26813 catalog_manager.cc:4841] TS b48b41ec43a24816bd0bc1468caed0a6 (127.28.251.195:37241): tablet 933e82e027614ebbb2c57174a359e1e9 (table test-table [id=9dae0388828941c9973d5208e66e7977]) successfully deleted
I20260808 04:02:33.870201 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 26947
I20260808 04:02:33.907601 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 27160
I20260808 04:02:33.916548 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:33.908523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.28.251.194:41197)
0808 04:02:33.908717 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:33.908723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:33.908749 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:02:33.908964 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:33.908970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:33.909093 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:33.909200 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:33.909663 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.910907 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.912791 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:33.913478 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:33.913620 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:33.914910 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:33.914921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:33.914939 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:33.915774 (+ 835us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:33.915850 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:33.915877 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:33.915998 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:33.916142 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 04:02:33.916239 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:02:33.917361 27378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:34.008693 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 27160
I20260808 04:02:34.036425 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 27391
I20260808 04:02:34.044011 27640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:34.037283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.28.251.195:37241)
0808 04:02:34.037488 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:34.037494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:34.037589 (+ 95us) server_negotiation.cc:407] Connection header received
0808 04:02:34.037795 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:34.037801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:34.037906 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:34.037995 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:34.038460 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:34.039288 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:34.040672 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:34.041153 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:34.041275 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:34.042443 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:34.042456 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:34.042474 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:34.043146 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:34.043215 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:34.043238 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:34.043367 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:34.043526 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 04:02:34.043648 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:02:34.044862 27596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:34.172963 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 27391
I20260808 04:02:34.210403 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 26793
I20260808 04:02:34.211638 26888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:34.322947 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 26793
2026-08-08T04:02:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3092 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:02:34 dist-test-slave-pww0 krb5kdc[27703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:02:34 dist-test-slave-pww0 krb5kdc[27703](info): set up 2 sockets
Aug 08 04:02:34 dist-test-slave-pww0 krb5kdc[27703](info): commencing operation
krb5kdc: starting...
W20260808 04:02:35.657678 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.282s user 0.003s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:02:35 dist-test-slave-pww0 krb5kdc[27703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:02:36.028214 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43281
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:36.406388 27723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:36.406726 27723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:36.406862 27723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:36.414563 27723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:02:36.414695 27723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:36.414779 27723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:36.414857 27723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:02:36.414918 27723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:02:36.425437 27723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:36.426896 27723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:36.428381 27723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:36.440861 27733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:36.440963 27737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:36.440984 27735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:36.442255 27723 server_base.cc:1034] running on GCE node
I20260808 04:02:36.443161 27723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:36.444842 27723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:36.446060 27723 hybrid_clock.cc:648] HybridClock initialized: now 1786161756446000 us; error 47 us; skew 500 ppm
I20260808 04:02:36.446455 27723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:36 dist-test-slave-pww0 krb5kdc[27703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161756, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:36.449847 27723 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:02:36.451635 27723 webserver.cc:460] Webserver started at http://127.28.251.254:39735/ using document root <none> and password file <none>
I20260808 04:02:36.452405 27723 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:36.452603 27723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:36.453003 27723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:36.456681 27723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "70c9eb85bfb84fd18b213ae9bd25addc"
format_stamp: "Formatted at 2026-08-08 04:02:36 on dist-test-slave-pww0"
server_key: "146f0623e221b2285e230393271e5ff7"
server_key_iv: "c15aa105871a9757fa2c54ac56bdb1f1"
server_key_version: "encryptionkey@0"
I20260808 04:02:36.457584 27723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "70c9eb85bfb84fd18b213ae9bd25addc"
format_stamp: "Formatted at 2026-08-08 04:02:36 on dist-test-slave-pww0"
server_key: "146f0623e221b2285e230393271e5ff7"
server_key_iv: "c15aa105871a9757fa2c54ac56bdb1f1"
server_key_version: "encryptionkey@0"
I20260808 04:02:36.464586 27723 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260808 04:02:36.468737 27747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:36.470288 27723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:02:36.470510 27723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "70c9eb85bfb84fd18b213ae9bd25addc"
format_stamp: "Formatted at 2026-08-08 04:02:36 on dist-test-slave-pww0"
server_key: "146f0623e221b2285e230393271e5ff7"
server_key_iv: "c15aa105871a9757fa2c54ac56bdb1f1"
server_key_version: "encryptionkey@0"
I20260808 04:02:36.470726 27723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:36.492875 27723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:36.498024 27723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:36.498345 27723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:36.517463 27723 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:43281
I20260808 04:02:36.517495 27836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:43281 every 8 connection(s)
I20260808 04:02:36.519018 27723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:02:36.523424 27839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:36.526371 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 27723
I20260808 04:02:36.526566 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:02:36.527094 29679 external_mini_cluster.cc:1428] Setting key 3e452c09c80b9802740929b90d3475dd
I20260808 04:02:36.536725 27839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc: Bootstrap starting.
I20260808 04:02:36.540992 27839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:36.542462 27839 log.cc:824] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:02:36 dist-test-slave-pww0 krb5kdc[27703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:36.546088 27839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc: No bootstrap required, opened a new log
I20260808 04:02:36.553047 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:36.529236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.28.251.254:43281)
0808 04:02:36.530073 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:36.530087 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:36.530135 (+ 48us) server_negotiation.cc:407] Connection header received
0808 04:02:36.532021 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:36.532050 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:36.532527 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:36.532936 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:36.534999 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:36.536570 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:36.538569 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:36.539374 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:36.539595 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:36.545023 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:36.545032 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:36.545048 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:36.545090 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:36.548440 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:36.549292 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:36.549299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:36.549306 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:36.549507 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:36.549933 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:36.549937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:36.549939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:36.550565 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:36.550889 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:36.551390 (+ 501us) server_negotiation.cc:299] Negotiation successful
0808 04:02:36.551889 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":369,"threads_started":1}
I20260808 04:02:36.554492 27839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c9eb85bfb84fd18b213ae9bd25addc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43281 } }
I20260808 04:02:36.554903 27839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:36.555002 27839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c9eb85bfb84fd18b213ae9bd25addc, State: Initialized, Role: FOLLOWER
I20260808 04:02:36.555660 27839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c9eb85bfb84fd18b213ae9bd25addc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43281 } }
I20260808 04:02:36.555922 27839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:02:36.556100 27839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:02:36.556262 27839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:36.558517 27839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c9eb85bfb84fd18b213ae9bd25addc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43281 } }
I20260808 04:02:36.559046 27839 leader_election.cc:302] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70c9eb85bfb84fd18b213ae9bd25addc; no voters:
I20260808 04:02:36.559708 27839 leader_election.cc:288] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:02:36.559863 27856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:36.560529 27856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [term 1 LEADER]: Becoming Leader. State: Replica: 70c9eb85bfb84fd18b213ae9bd25addc, State: Running, Role: LEADER
I20260808 04:02:36.561190 27856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c9eb85bfb84fd18b213ae9bd25addc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43281 } }
I20260808 04:02:36.562139 27839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:02:36.564520 27859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70c9eb85bfb84fd18b213ae9bd25addc. Latest consensus state: current_term: 1 leader_uuid: "70c9eb85bfb84fd18b213ae9bd25addc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c9eb85bfb84fd18b213ae9bd25addc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43281 } } }
I20260808 04:02:36.565002 27859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:36.564592 27857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70c9eb85bfb84fd18b213ae9bd25addc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c9eb85bfb84fd18b213ae9bd25addc" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43281 } } }
I20260808 04:02:36.565409 27857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:36.568095 27870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:02:36.572712 27870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:02:36.582388 27870 catalog_manager.cc:1347] Generated new cluster ID: e9011a26d21445809c02807410aa8ecf
I20260808 04:02:36.582520 27870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:02:36.620005 27870 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:02:36.622092 27870 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:02:36.639111 27870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc: Generated new TSK 0
I20260808 04:02:36.640420 27870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 04:02:36.723810 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43281
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:37.093930 27893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:37.094256 27893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:37.094383 27893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:37.101979 27893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:37.102109 27893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:37.102257 27893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:02:37.112641 27893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:37.114087 27893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:37.115551 27893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:37.132400 27905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:37.132352 27908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:37.132715 27902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:37.133559 27893 server_base.cc:1034] running on GCE node
I20260808 04:02:37.134248 27893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:37.135380 27893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:37.136581 27893 hybrid_clock.cc:648] HybridClock initialized: now 1786161757136515 us; error 53 us; skew 500 ppm
I20260808 04:02:37.136924 27893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:37 dist-test-slave-pww0 krb5kdc[27703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161757, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:37.140370 27893 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 04:02:37.141995 27893 webserver.cc:460] Webserver started at http://127.28.251.193:33211/ using document root <none> and password file <none>
I20260808 04:02:37.142740 27893 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:37.142868 27893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:37.143189 27893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:37.146389 27893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "63c772f910f84979bee8f15e7827c762"
format_stamp: "Formatted at 2026-08-08 04:02:37 on dist-test-slave-pww0"
server_key: "1fb8935a1214d8898b51ca502c01ba9f"
server_key_iv: "e615296b6d7bb2682b2c84aed7cba35b"
server_key_version: "encryptionkey@0"
I20260808 04:02:37.147231 27893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "63c772f910f84979bee8f15e7827c762"
format_stamp: "Formatted at 2026-08-08 04:02:37 on dist-test-slave-pww0"
server_key: "1fb8935a1214d8898b51ca502c01ba9f"
server_key_iv: "e615296b6d7bb2682b2c84aed7cba35b"
server_key_version: "encryptionkey@0"
I20260808 04:02:37.153014 27893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:02:37.156996 27917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:37.158587 27893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:02:37.158805 27893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "63c772f910f84979bee8f15e7827c762"
format_stamp: "Formatted at 2026-08-08 04:02:37 on dist-test-slave-pww0"
server_key: "1fb8935a1214d8898b51ca502c01ba9f"
server_key_iv: "e615296b6d7bb2682b2c84aed7cba35b"
server_key_version: "encryptionkey@0"
I20260808 04:02:37.159025 27893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:37.179080 27893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:37.184387 27893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:37.184710 27893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:37.185860 27893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:37.187418 27893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:37.187536 27893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 04:02:37.187676 27893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:37.187733 27893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:37.220480 27893 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:45789
I20260808 04:02:37.220521 28114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:45789 every 8 connection(s)
I20260808 04:02:37.221854 27893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:02:37.231544 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 27893
I20260808 04:02:37.231796 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 04:02:37.232301 29679 external_mini_cluster.cc:1428] Setting key 3592b970383ef2a3a17be07a062b90b5
Aug 08 04:02:37 dist-test-slave-pww0 krb5kdc[27703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161757, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:37.244979 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:37.224579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:44011 (local address 127.28.251.254:43281)
0808 04:02:37.225056 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:37.225061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:37.226013 (+ 952us) server_negotiation.cc:407] Connection header received
0808 04:02:37.228012 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:37.228018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:37.228105 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:37.228211 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:37.229913 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:37.230682 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:37.232560 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:37.233049 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:37.233198 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:37.237826 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:37.237833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:37.237838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:37.237887 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:37.241250 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:37.242175 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:37.242183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:37.242186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:37.242371 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:37.242830 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:37.242835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:37.242839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:37.243251 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:37.243414 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:37.244302 (+ 888us) server_negotiation.cc:299] Negotiation successful
0808 04:02:37.244576 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":251,"threads_started":1}
I20260808 04:02:37.245915 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:37.225064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43281 (local address 127.28.251.193:44011)
0808 04:02:37.225833 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:37.225875 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:37.227702 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:37.228505 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:37.228514 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:37.229084 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:37.229709 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:37.229729 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:37.230873 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:37.230878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:37.232391 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:37.232410 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:37.233364 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:37.233374 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:37.233688 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:37.234497 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:02:37.234546 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:37.237520 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:37.241448 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:37.241457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:37.241474 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:37.241993 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:37.242479 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:37.242482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:37.242485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:37.242700 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:37.243400 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:37.243407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:37.243855 (+ 448us) client_negotiation.cc:769] Sending connection context
0808 04:02:37.244110 (+ 255us) client_negotiation.cc:240] Negotiation successful
0808 04:02:37.244574 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":312,"threads_started":1}
I20260808 04:02:37.247635 28116 heartbeater.cc:347] Connected to a master server at 127.28.251.254:43281
I20260808 04:02:37.248023 28116 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:37.249022 28116 heartbeater.cc:511] Master 127.28.251.254:43281 requested a full tablet report, sending...
I20260808 04:02:37.251597 27770 ts_manager.cc:194] Registered new tserver with Master: 63c772f910f84979bee8f15e7827c762 (127.28.251.193:45789)
I20260808 04:02:37.253886 27770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:44011
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 04:02:37.315215 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43281
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:37.691905 28126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:37.692237 28126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:37.692384 28126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:37.699913 28126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:37.700068 28126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:37.700237 28126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:02:37.710587 28126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:37.712049 28126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:37.713416 28126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:37.727546 28143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:37.727504 28140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:37.727699 28145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:37.728157 28126 server_base.cc:1034] running on GCE node
I20260808 04:02:37.729017 28126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:37.730060 28126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:37.731333 28126 hybrid_clock.cc:648] HybridClock initialized: now 1786161757731250 us; error 55 us; skew 500 ppm
I20260808 04:02:37.731794 28126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:37 dist-test-slave-pww0 krb5kdc[27703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161757, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:37.735189 28126 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 04:02:37.736972 28126 webserver.cc:460] Webserver started at http://127.28.251.194:46115/ using document root <none> and password file <none>
I20260808 04:02:37.737713 28126 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:37.737829 28126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:37.738130 28126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:37.741429 28126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "0b8a5d3553464b2cbeeb57f7badcffd6"
format_stamp: "Formatted at 2026-08-08 04:02:37 on dist-test-slave-pww0"
server_key: "ca62031b7519efe477182402b241a798"
server_key_iv: "b703945de62cccc394291d8b22d0744e"
server_key_version: "encryptionkey@0"
I20260808 04:02:37.742278 28126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "0b8a5d3553464b2cbeeb57f7badcffd6"
format_stamp: "Formatted at 2026-08-08 04:02:37 on dist-test-slave-pww0"
server_key: "ca62031b7519efe477182402b241a798"
server_key_iv: "b703945de62cccc394291d8b22d0744e"
server_key_version: "encryptionkey@0"
I20260808 04:02:37.747852 28126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 04:02:37.751621 28164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:37.753211 28126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:02:37.753408 28126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "0b8a5d3553464b2cbeeb57f7badcffd6"
format_stamp: "Formatted at 2026-08-08 04:02:37 on dist-test-slave-pww0"
server_key: "ca62031b7519efe477182402b241a798"
server_key_iv: "b703945de62cccc394291d8b22d0744e"
server_key_version: "encryptionkey@0"
I20260808 04:02:37.753628 28126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:37.772786 28126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:37.777642 28126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:37.777938 28126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:37.779058 28126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:37.780573 28126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:37.780658 28126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:37.780783 28126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:37.780838 28126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:37.813529 28126 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:41697
I20260808 04:02:37.813552 28363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:41697 every 8 connection(s)
I20260808 04:02:37.814899 28126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:02:37.823849 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 28126
I20260808 04:02:37.824076 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 04:02:37.824548 29679 external_mini_cluster.cc:1428] Setting key e04829315f33c5ce5d320e28986b8db2
Aug 08 04:02:37 dist-test-slave-pww0 krb5kdc[27703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161757, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:37.838269 28367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:37.817611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:50519 (local address 127.28.251.254:43281)
0808 04:02:37.818106 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:37.818111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:37.819165 (+ 1054us) server_negotiation.cc:407] Connection header received
0808 04:02:37.821099 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:37.821104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:37.821195 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:37.821298 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:37.822929 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:37.823891 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:37.825783 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:37.826485 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:37.826620 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:37.830875 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:37.830881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:37.830884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:37.830919 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:37.833588 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:37.834506 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:37.834527 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:37.834530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:37.834715 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:37.835346 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:37.835352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:37.835356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:37.835796 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:37.835957 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:37.837149 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0808 04:02:37.837970 (+ 821us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260808 04:02:37.839047 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:37.818197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43281 (local address 127.28.251.194:50519)
0808 04:02:37.818973 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:37.819014 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:37.820803 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:37.821572 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:37.821581 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:37.822120 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:37.822737 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:37.822756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:37.824105 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:37.824109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:37.825628 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:37.825642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:37.826745 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:37.826753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:37.827060 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:37.827879 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:02:37.827916 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:37.830607 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:37.833787 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:37.833798 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:37.833815 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:37.834327 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:37.834875 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:37.834879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:37.834883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:37.835208 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:37.836008 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:37.836017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:37.836536 (+ 519us) client_negotiation.cc:769] Sending connection context
0808 04:02:37.836900 (+ 364us) client_negotiation.cc:240] Negotiation successful
0808 04:02:37.837531 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":310,"threads_started":1}
I20260808 04:02:37.840703 28364 heartbeater.cc:347] Connected to a master server at 127.28.251.254:43281
I20260808 04:02:37.841080 28364 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:37.842058 28364 heartbeater.cc:511] Master 127.28.251.254:43281 requested a full tablet report, sending...
I20260808 04:02:37.843967 27770 ts_manager.cc:194] Registered new tserver with Master: 0b8a5d3553464b2cbeeb57f7badcffd6 (127.28.251.194:41697)
I20260808 04:02:37.844843 27770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:50519
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 04:02:37.903801 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43281
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:02:38.257884 28116 heartbeater.cc:503] Master 127.28.251.254:43281 was elected leader, sending a full tablet report...
W20260808 04:02:38.272958 28377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:38.273315 28377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:38.273466 28377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:38.281477 28377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:38.281641 28377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:38.281816 28377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:02:38.292330 28377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:43281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:38.293776 28377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:38.295167 28377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:38.309350 28386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:38.309268 28388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:38.309294 28390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:38.310467 28377 server_base.cc:1034] running on GCE node
I20260808 04:02:38.311165 28377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:38.312309 28377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:38.313529 28377 hybrid_clock.cc:648] HybridClock initialized: now 1786161758313463 us; error 58 us; skew 500 ppm
I20260808 04:02:38.313876 28377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:38 dist-test-slave-pww0 krb5kdc[27703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161758, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:38.317180 28377 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 04:02:38.318701 28377 webserver.cc:460] Webserver started at http://127.28.251.195:35485/ using document root <none> and password file <none>
I20260808 04:02:38.319427 28377 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:38.319603 28377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:38.319943 28377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:38.323180 28377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "ed068923ee384ce59a25dc355b55b65e"
format_stamp: "Formatted at 2026-08-08 04:02:38 on dist-test-slave-pww0"
server_key: "57899a1490e626e853023525e6a0c202"
server_key_iv: "6abdf4f5fc037d8b6c1ab48792500592"
server_key_version: "encryptionkey@0"
I20260808 04:02:38.324082 28377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "ed068923ee384ce59a25dc355b55b65e"
format_stamp: "Formatted at 2026-08-08 04:02:38 on dist-test-slave-pww0"
server_key: "57899a1490e626e853023525e6a0c202"
server_key_iv: "6abdf4f5fc037d8b6c1ab48792500592"
server_key_version: "encryptionkey@0"
I20260808 04:02:38.329922 28377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:02:38.334074 28401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:38.335739 28377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:02:38.335937 28377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "ed068923ee384ce59a25dc355b55b65e"
format_stamp: "Formatted at 2026-08-08 04:02:38 on dist-test-slave-pww0"
server_key: "57899a1490e626e853023525e6a0c202"
server_key_iv: "6abdf4f5fc037d8b6c1ab48792500592"
server_key_version: "encryptionkey@0"
I20260808 04:02:38.336184 28377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:38.353995 28377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:38.358512 28377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:38.358836 28377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:38.360116 28377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:38.361775 28377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:38.361913 28377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:38.362059 28377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:38.362114 28377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:38.398021 28377 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:40437
I20260808 04:02:38.398082 28572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:40437 every 8 connection(s)
I20260808 04:02:38.399545 28377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:02:38.402046 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 28377
I20260808 04:02:38.402212 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 04:02:38.402693 29679 external_mini_cluster.cc:1428] Setting key 7da3b03ebacc0cc279281f0fcc8ae828
Aug 08 04:02:38 dist-test-slave-pww0 krb5kdc[27703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161758, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:38.421633 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.402540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:53487 (local address 127.28.251.254:43281)
0808 04:02:38.403074 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:38.403080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:38.404207 (+ 1127us) server_negotiation.cc:407] Connection header received
0808 04:02:38.406457 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:38.406461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:38.406534 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:38.406626 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:38.408316 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.409179 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.410891 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.411309 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.411415 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.415133 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:38.415138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:38.415142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:38.415178 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:38.418270 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:38.419203 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:38.419208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:38.419210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:38.419370 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:38.419964 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:38.419968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:38.419971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:38.420265 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:38.420397 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:38.421142 (+ 745us) server_negotiation.cc:299] Negotiation successful
0808 04:02:38.421330 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":241,"threads_started":1}
I20260808 04:02:38.422673 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.403113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43281 (local address 127.28.251.195:53487)
0808 04:02:38.404033 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:38.404084 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:38.406197 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:38.406836 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:38.406844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:38.407451 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:38.408117 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.408136 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.409369 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.409373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.410753 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.410766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.411584 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.411591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.411815 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.412497 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:02:38.412522 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:38.414879 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:38.418446 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:38.418459 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:38.418479 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:38.419059 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:38.419563 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:38.419567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:38.419570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:38.419829 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:38.420377 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:38.420383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:38.420765 (+ 382us) client_negotiation.cc:769] Sending connection context
0808 04:02:38.421044 (+ 279us) client_negotiation.cc:240] Negotiation successful
0808 04:02:38.421487 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":390,"threads_started":1}
I20260808 04:02:38.424171 28573 heartbeater.cc:347] Connected to a master server at 127.28.251.254:43281
I20260808 04:02:38.424513 28573 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:38.425330 28573 heartbeater.cc:511] Master 127.28.251.254:43281 requested a full tablet report, sending...
I20260808 04:02:38.426925 27770 ts_manager.cc:194] Registered new tserver with Master: ed068923ee384ce59a25dc355b55b65e (127.28.251.195:40437)
I20260808 04:02:38.427841 27770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:53487
I20260808 04:02:38.431358 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:02:38.450199 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.435754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.28.251.254:43281)
0808 04:02:38.435943 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:38.435947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:38.436510 (+ 563us) server_negotiation.cc:407] Connection header received
0808 04:02:38.436687 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:38.436691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:38.436778 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:38.436883 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:38.438461 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.439226 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.440692 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.441158 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.441282 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.444356 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:38.444362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:38.444365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:38.444405 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:38.447001 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:38.447977 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:38.447981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:38.447983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:38.448119 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:38.448637 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:38.448641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:38.448643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:38.448992 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:38.449266 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:38.449707 (+ 441us) server_negotiation.cc:299] Negotiation successful
0808 04:02:38.449919 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:02:38.454944 27770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:02:38.463032 27770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:02:38.487054 28606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.477155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.28.251.193:45789)
0808 04:02:38.478851 (+ 1696us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:38.478858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:38.478881 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:02:38.478960 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:38.478966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:38.479204 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:38.480162 (+ 958us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:38.480373 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.481393 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.484314 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.485770 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.486000 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.486189 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:38.486516 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 04:02:38.486702 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1538,"thread_start_us":1360,"threads_started":1}
I20260808 04:02:38.487071 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.476853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:45789 (local address 127.0.0.1:52820)
0808 04:02:38.477926 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:38.477981 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:38.478172 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:38.479512 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:38.479520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:38.479555 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:38.479976 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.479988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.481538 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.481544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.484122 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.484138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.486018 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.486026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.486255 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.486315 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 04:02:38.486467 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 04:02:38.486668 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":622,"threads_started":1}
I20260808 04:02:38.491510 27999 tablet_service.cc:1467] Processing CreateTablet for tablet 9facf13d45f045e889cb1995b61bb9ea (DEFAULT_TABLE table=test-table [id=56a6262eb1c143b7b36a52fa274d8abd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:38.492247 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.478317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:40437 (local address 127.0.0.1:53378)
0808 04:02:38.484078 (+ 5761us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:38.484097 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:38.484217 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:38.484933 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:38.484938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:38.484965 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:38.485322 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.485342 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.487234 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.487239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.489140 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.489155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.491288 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.491295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.491493 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.491528 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:02:38.491624 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 04:02:38.491779 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5603,"mutex_wait_us":53,"thread_start_us":3359,"threads_started":1}
I20260808 04:02:38.492859 28610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.477993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.28.251.194:41697)
0808 04:02:38.482740 (+ 4747us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:38.482749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:38.482779 (+ 30us) server_negotiation.cc:407] Connection header received
0808 04:02:38.482879 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:38.482886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:38.483150 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:38.483364 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:38.484281 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.485628 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.489676 (+ 4048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.491161 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.491339 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.491541 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:38.492382 (+ 841us) server_negotiation.cc:299] Negotiation successful
0808 04:02:38.492575 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4579,"thread_start_us":4471,"threads_started":1}
I20260808 04:02:38.492998 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.478909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.28.251.195:40437)
0808 04:02:38.479705 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:38.479714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:38.484369 (+ 4655us) server_negotiation.cc:407] Connection header received
0808 04:02:38.484487 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:38.484495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:38.484717 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:38.484923 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:38.485528 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.487052 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.489325 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.490710 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.490928 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.491124 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:38.492501 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0808 04:02:38.492658 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":415,"threads_started":1}
I20260808 04:02:38.493331 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.477492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:41697 (local address 127.0.0.1:48524)
0808 04:02:38.481993 (+ 4501us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:38.482012 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:38.482131 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:38.483392 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:38.483398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:38.483433 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:38.484000 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.484014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.485753 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.485759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.489467 (+ 3708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.489484 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.491691 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.491697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.491886 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.491922 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 04:02:38.491995 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 04:02:38.492130 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4361,"mutex_wait_us":310,"thread_start_us":6218,"threads_started":1}
I20260808 04:02:38.494372 27999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9facf13d45f045e889cb1995b61bb9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:38.496953 28470 tablet_service.cc:1467] Processing CreateTablet for tablet 9facf13d45f045e889cb1995b61bb9ea (DEFAULT_TABLE table=test-table [id=56a6262eb1c143b7b36a52fa274d8abd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:38.497079 28260 tablet_service.cc:1467] Processing CreateTablet for tablet 9facf13d45f045e889cb1995b61bb9ea (DEFAULT_TABLE table=test-table [id=56a6262eb1c143b7b36a52fa274d8abd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:38.498306 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9facf13d45f045e889cb1995b61bb9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:38.498553 28260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9facf13d45f045e889cb1995b61bb9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:38.510251 28619 tablet_bootstrap.cc:492] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762: Bootstrap starting.
I20260808 04:02:38.513285 28622 tablet_bootstrap.cc:492] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: Bootstrap starting.
I20260808 04:02:38.514992 28621 tablet_bootstrap.cc:492] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: Bootstrap starting.
I20260808 04:02:38.515271 28619 tablet_bootstrap.cc:654] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:38.516690 28619 log.cc:824] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:38.518174 28622 tablet_bootstrap.cc:654] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:38.519251 28622 log.cc:824] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:38.519384 28621 tablet_bootstrap.cc:654] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:38.520596 28621 log.cc:824] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:38.521543 28619 tablet_bootstrap.cc:492] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762: No bootstrap required, opened a new log
I20260808 04:02:38.521916 28619 ts_tablet_manager.cc:1397] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260808 04:02:38.525771 28621 tablet_bootstrap.cc:492] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: No bootstrap required, opened a new log
I20260808 04:02:38.526103 28621 ts_tablet_manager.cc:1397] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260808 04:02:38.528009 28622 tablet_bootstrap.cc:492] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: No bootstrap required, opened a new log
I20260808 04:02:38.528406 28622 ts_tablet_manager.cc:1397] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260808 04:02:38.531726 28619 raft_consensus.cc:348] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c772f910f84979bee8f15e7827c762" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45789 } } peers { permanent_uuid: "ed068923ee384ce59a25dc355b55b65e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 40437 } } peers { permanent_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41697 } }
I20260808 04:02:38.532495 28619 raft_consensus.cc:374] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:38.532671 28619 raft_consensus.cc:729] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c772f910f84979bee8f15e7827c762, State: Initialized, Role: FOLLOWER
I20260808 04:02:38.533653 28619 consensus_queue.cc:260] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c772f910f84979bee8f15e7827c762" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45789 } } peers { permanent_uuid: "ed068923ee384ce59a25dc355b55b65e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 40437 } } peers { permanent_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41697 } }
I20260808 04:02:38.540118 28619 ts_tablet_manager.cc:1428] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260808 04:02:38.539921 28621 raft_consensus.cc:348] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c772f910f84979bee8f15e7827c762" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45789 } } peers { permanent_uuid: "ed068923ee384ce59a25dc355b55b65e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 40437 } } peers { permanent_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41697 } }
I20260808 04:02:38.540582 28621 raft_consensus.cc:374] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:38.540740 28621 raft_consensus.cc:729] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed068923ee384ce59a25dc355b55b65e, State: Initialized, Role: FOLLOWER
I20260808 04:02:38.540874 28622 raft_consensus.cc:348] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c772f910f84979bee8f15e7827c762" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45789 } } peers { permanent_uuid: "ed068923ee384ce59a25dc355b55b65e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 40437 } } peers { permanent_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41697 } }
I20260808 04:02:38.541388 28622 raft_consensus.cc:374] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:38.541498 28622 raft_consensus.cc:729] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8a5d3553464b2cbeeb57f7badcffd6, State: Initialized, Role: FOLLOWER
I20260808 04:02:38.541700 28621 consensus_queue.cc:260] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c772f910f84979bee8f15e7827c762" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45789 } } peers { permanent_uuid: "ed068923ee384ce59a25dc355b55b65e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 40437 } } peers { permanent_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41697 } }
I20260808 04:02:38.542085 28622 consensus_queue.cc:260] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c772f910f84979bee8f15e7827c762" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45789 } } peers { permanent_uuid: "ed068923ee384ce59a25dc355b55b65e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 40437 } } peers { permanent_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41697 } }
I20260808 04:02:38.543422 28364 heartbeater.cc:503] Master 127.28.251.254:43281 was elected leader, sending a full tablet report...
I20260808 04:02:38.543720 28621 ts_tablet_manager.cc:1428] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260808 04:02:38.543872 28573 heartbeater.cc:503] Master 127.28.251.254:43281 was elected leader, sending a full tablet report...
I20260808 04:02:38.544094 28622 ts_tablet_manager.cc:1428] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260808 04:02:38.566969 28366 tablet.cc:2369] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:02:38.626878 28632 raft_consensus.cc:482] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:38.627240 28632 raft_consensus.cc:504] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c772f910f84979bee8f15e7827c762" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45789 } } peers { permanent_uuid: "ed068923ee384ce59a25dc355b55b65e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 40437 } } peers { permanent_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41697 } }
I20260808 04:02:38.629194 28632 leader_election.cc:288] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63c772f910f84979bee8f15e7827c762 (127.28.251.193:45789), ed068923ee384ce59a25dc355b55b65e (127.28.251.195:40437)
I20260808 04:02:38.636739 28606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.629740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:57041 (local address 127.28.251.193:45789)
0808 04:02:38.629998 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:38.630005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:38.630254 (+ 249us) server_negotiation.cc:407] Connection header received
0808 04:02:38.630520 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:38.630527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:38.630615 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:38.630712 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:38.631501 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.632483 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.634677 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.635815 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.635957 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.636018 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:38.636404 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 04:02:38.636504 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 04:02:38.636933 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.629485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:45789 (local address 127.28.251.194:57041)
0808 04:02:38.630116 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:38.630138 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:38.630330 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:38.630807 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:38.630813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:38.630845 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:38.631240 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.631255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.632620 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.632626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.634483 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.634497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.636041 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.636048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.636228 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.636267 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 04:02:38.636389 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 04:02:38.636542 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":335,"threads_started":1}
I20260808 04:02:38.637665 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.630244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:40437 (local address 127.28.251.194:52929)
0808 04:02:38.630950 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:38.630964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:38.631074 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:38.631501 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:38.631506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:38.631536 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:38.631984 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.632006 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.633568 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.633573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.635331 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:38.635349 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.636912 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.636922 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:38.637120 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.637164 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 04:02:38.637288 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 04:02:38.637417 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":357,"threads_started":1}
I20260808 04:02:38.637748 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.630489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:52929 (local address 127.28.251.195:40437)
0808 04:02:38.630708 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:38.630713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:38.631068 (+ 355us) server_negotiation.cc:407] Connection header received
0808 04:02:38.631180 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:38.631185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:38.631264 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:38.631375 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:38.632219 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.633359 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.635569 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.636646 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.636806 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.636893 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:38.637364 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 04:02:38.637507 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 04:02:38.638361 28031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9facf13d45f045e889cb1995b61bb9ea" candidate_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c772f910f84979bee8f15e7827c762" is_pre_election: true
I20260808 04:02:38.638770 28500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9facf13d45f045e889cb1995b61bb9ea" candidate_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed068923ee384ce59a25dc355b55b65e" is_pre_election: true
I20260808 04:02:38.638939 28031 raft_consensus.cc:2454] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8a5d3553464b2cbeeb57f7badcffd6 in term 0.
I20260808 04:02:38.639317 28500 raft_consensus.cc:2454] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8a5d3553464b2cbeeb57f7badcffd6 in term 0.
I20260808 04:02:38.639808 28175 leader_election.cc:302] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8a5d3553464b2cbeeb57f7badcffd6, 63c772f910f84979bee8f15e7827c762; no voters:
I20260808 04:02:38.640261 28632 raft_consensus.cc:2781] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:02:38.640499 28632 raft_consensus.cc:482] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:02:38.640633 28632 raft_consensus.cc:3037] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:38.643261 28632 raft_consensus.cc:504] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c772f910f84979bee8f15e7827c762" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45789 } } peers { permanent_uuid: "ed068923ee384ce59a25dc355b55b65e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 40437 } } peers { permanent_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41697 } }
I20260808 04:02:38.643961 28632 leader_election.cc:288] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [CANDIDATE]: Term 1 election: Requested vote from peers 63c772f910f84979bee8f15e7827c762 (127.28.251.193:45789), ed068923ee384ce59a25dc355b55b65e (127.28.251.195:40437)
I20260808 04:02:38.644657 28031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9facf13d45f045e889cb1995b61bb9ea" candidate_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c772f910f84979bee8f15e7827c762"
I20260808 04:02:38.644662 28500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9facf13d45f045e889cb1995b61bb9ea" candidate_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed068923ee384ce59a25dc355b55b65e"
I20260808 04:02:38.644878 28031 raft_consensus.cc:3037] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:38.644909 28500 raft_consensus.cc:3037] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:38.647341 28500 raft_consensus.cc:2454] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8a5d3553464b2cbeeb57f7badcffd6 in term 1.
I20260808 04:02:38.647334 28031 raft_consensus.cc:2454] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8a5d3553464b2cbeeb57f7badcffd6 in term 1.
I20260808 04:02:38.647944 28175 leader_election.cc:302] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8a5d3553464b2cbeeb57f7badcffd6, 63c772f910f84979bee8f15e7827c762; no voters:
I20260808 04:02:38.648384 28632 raft_consensus.cc:2781] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:38.648942 28632 raft_consensus.cc:686] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 1 LEADER]: Becoming Leader. State: Replica: 0b8a5d3553464b2cbeeb57f7badcffd6, State: Running, Role: LEADER
I20260808 04:02:38.649542 28632 consensus_queue.cc:237] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c772f910f84979bee8f15e7827c762" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45789 } } peers { permanent_uuid: "ed068923ee384ce59a25dc355b55b65e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 40437 } } peers { permanent_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41697 } }
W20260808 04:02:38.651286 28575 tablet.cc:2369] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:02:38.654667 27769 catalog_manager.cc:5495] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b8a5d3553464b2cbeeb57f7badcffd6 (127.28.251.194). New cstate: current_term: 1 leader_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c772f910f84979bee8f15e7827c762" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed068923ee384ce59a25dc355b55b65e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 40437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b8a5d3553464b2cbeeb57f7badcffd6" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 41697 } health_report { overall_health: HEALTHY } } }
I20260808 04:02:38.692847 28610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.687057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.28.251.194:41697)
0808 04:02:38.687300 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:38.687306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:38.687346 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:02:38.687564 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:38.687568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:38.687654 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:38.687797 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:02:38.688252 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.688983 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.690415 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.690830 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.690944 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.691657 (+ 713us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:02:38.692186 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:02:38.692380 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:38.692536 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 04:02:38.692588 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 04:02:38.702018 28031 raft_consensus.cc:1261] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762 [term 1 FOLLOWER]: Refusing update from remote peer 0b8a5d3553464b2cbeeb57f7badcffd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:38.702044 28500 raft_consensus.cc:1261] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e [term 1 FOLLOWER]: Refusing update from remote peer 0b8a5d3553464b2cbeeb57f7badcffd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:38.703099 28639 consensus_queue.cc:1035] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c772f910f84979bee8f15e7827c762" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:02:38.703534 28632 consensus_queue.cc:1035] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed068923ee384ce59a25dc355b55b65e" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 40437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:02:38.714526 28646 mvcc.cc:205] Tried to move back new op lower bound from 7316118563631738880 to 7316118563433107456. Current Snapshot: MvccSnapshot[applied={T|T < 7316118563631738880}]
I20260808 04:02:38.717772 28644 mvcc.cc:205] Tried to move back new op lower bound from 7316118563631738880 to 7316118563433107456. Current Snapshot: MvccSnapshot[applied={T|T < 7316118563631738880}]
I20260808 04:02:38.720953 28645 mvcc.cc:205] Tried to move back new op lower bound from 7316118563631738880 to 7316118563433107456. Current Snapshot: MvccSnapshot[applied={T|T < 7316118563631738880}]
I20260808 04:02:38.737802 27769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:38.738101 27769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:38.742069 27769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70c9eb85bfb84fd18b213ae9bd25addc: Sending DeleteTablet for 3 replicas of tablet 9facf13d45f045e889cb1995b61bb9ea
I20260808 04:02:38.743443 28260 tablet_service.cc:1514] Processing DeleteTablet for tablet 9facf13d45f045e889cb1995b61bb9ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:38 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:48524
I20260808 04:02:38.743526 27999 tablet_service.cc:1514] Processing DeleteTablet for tablet 9facf13d45f045e889cb1995b61bb9ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:38 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:52820
I20260808 04:02:38.743762 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet 9facf13d45f045e889cb1995b61bb9ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:38 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:53378
I20260808 04:02:38.744700 28662 tablet_replica.cc:330] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: stopping tablet replica
I20260808 04:02:38.745242 28662 raft_consensus.cc:2229] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:38.745390 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 27893
I20260808 04:02:38.745873 28662 raft_consensus.cc:2258] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:38.748220 28662 ts_tablet_manager.cc:1905] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:38.755748 28657 tablet_replica.cc:330] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: stopping tablet replica
I20260808 04:02:38.756165 28662 ts_tablet_manager.cc:1918] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:38.756453 28662 log.cc:1196] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/9facf13d45f045e889cb1995b61bb9ea
I20260808 04:02:38.756618 28657 raft_consensus.cc:2229] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:02:38.757162 28662 ts_tablet_manager.cc:1939] T 9facf13d45f045e889cb1995b61bb9ea P ed068923ee384ce59a25dc355b55b65e: Deleting consensus metadata
I20260808 04:02:38.757650 28657 raft_consensus.cc:2258] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:38.759402 27752 catalog_manager.cc:4841] TS ed068923ee384ce59a25dc355b55b65e (127.28.251.195:40437): tablet 9facf13d45f045e889cb1995b61bb9ea (table test-table [id=56a6262eb1c143b7b36a52fa274d8abd]) successfully deleted
I20260808 04:02:38.760563 28657 ts_tablet_manager.cc:1905] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:38.769404 28657 ts_tablet_manager.cc:1918] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:38.769658 28657 log.cc:1196] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/9facf13d45f045e889cb1995b61bb9ea
I20260808 04:02:38.770277 28657 ts_tablet_manager.cc:1939] T 9facf13d45f045e889cb1995b61bb9ea P 0b8a5d3553464b2cbeeb57f7badcffd6: Deleting consensus metadata
I20260808 04:02:38.772400 27748 catalog_manager.cc:4841] TS 0b8a5d3553464b2cbeeb57f7badcffd6 (127.28.251.194:41697): tablet 9facf13d45f045e889cb1995b61bb9ea (table test-table [id=56a6262eb1c143b7b36a52fa274d8abd]) successfully deleted
I20260808 04:02:38.772923 28660 tablet_replica.cc:330] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762: stopping tablet replica
I20260808 04:02:38.773346 28660 raft_consensus.cc:2229] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:38.773857 28660 raft_consensus.cc:2258] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:38.776424 28660 ts_tablet_manager.cc:1905] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:38.782140 28660 ts_tablet_manager.cc:1918] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:38.782303 28660 log.cc:1196] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/9facf13d45f045e889cb1995b61bb9ea
I20260808 04:02:38.782744 28660 ts_tablet_manager.cc:1939] T 9facf13d45f045e889cb1995b61bb9ea P 63c772f910f84979bee8f15e7827c762: Deleting consensus metadata
I20260808 04:02:38.784581 27752 catalog_manager.cc:4841] TS 63c772f910f84979bee8f15e7827c762 (127.28.251.193:45789): tablet 9facf13d45f045e889cb1995b61bb9ea (table test-table [id=56a6262eb1c143b7b36a52fa274d8abd]) successfully deleted
Aug 08 04:02:38 dist-test-slave-pww0 krb5kdc[27703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 04:02:38.794512 28606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.747593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.28.251.193:45789)
0808 04:02:38.747853 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:38.747860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:38.771633 (+ 23773us) server_negotiation.cc:407] Connection header received
0808 04:02:38.771862 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:38.771869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:38.771971 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:38.772091 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:38.774125 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.775357 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.777313 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.778029 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.778200 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.788558 (+ 10358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:38.788573 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:38.788589 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:38.788633 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:38.791579 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:38.792508 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:38.792516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:38.792518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:38.792664 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:38.793076 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:38.793079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:38.793081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:38.793449 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:38.793681 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:38.794035 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 04:02:38.794216 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260808 04:02:38.795362 28110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:38.905004 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 27893
I20260808 04:02:38.946451 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 28126
Aug 08 04:02:38 dist-test-slave-pww0 krb5kdc[27703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 04:02:38.962831 28610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:38.947366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.28.251.194:41697)
0808 04:02:38.947605 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:38.947611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:38.947633 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:02:38.947699 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:38.947705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:38.947773 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:38.947856 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:38.949170 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.950007 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.951333 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:38.951907 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:38.952029 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:38.956446 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:38.956461 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:38.956475 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:38.956536 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:38.959769 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:38.960524 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:38.960532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:38.960535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:38.960695 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:38.961108 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:38.961111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:38.961114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:38.961512 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:38.961975 (+ 463us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:38.962221 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 04:02:38.962467 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:02:38.963742 28359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:39.083130 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 28126
I20260808 04:02:39.119365 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 28377
Aug 08 04:02:39 dist-test-slave-pww0 krb5kdc[27703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 04:02:39.138810 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:39.120383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.28.251.195:40437)
0808 04:02:39.120617 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:39.120624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:39.120718 (+ 94us) server_negotiation.cc:407] Connection header received
0808 04:02:39.120949 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:39.120954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:39.121035 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:39.121137 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:39.122559 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:39.123295 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:39.125045 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:39.125478 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:39.125612 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:39.131103 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:39.131129 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:39.131153 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:39.131213 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:39.135043 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:39.136148 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:39.136160 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:39.136164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:39.136464 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:39.136919 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:39.136924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:39.136928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:39.137472 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:39.137810 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:39.138116 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 04:02:39.138440 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260808 04:02:39.139704 28566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:39.232945 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 28377
I20260808 04:02:39.258766 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 27723
I20260808 04:02:39.260048 27825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:39.373339 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 27723
2026-08-08T04:02:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5049 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T04:02:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:02:39Z Disabled control of system clock
I20260808 04:02:39.482848 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40635
--webserver_interface=127.28.251.254
--webserver_port=0
--builtin_ntp_servers=127.28.251.212:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:40635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:02:39.851316 28673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:39.851694 28673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:39.851795 28673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:39.859237 28673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:02:39.859355 28673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:39.859395 28673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:39.859488 28673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:02:39.859546 28673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:02:39.869156 28673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:40635
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:40635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:39.870543 28673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:39.872416 28673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:39.884438 28686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:39.884483 28682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:39.884459 28683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:39.885084 28673 server_base.cc:1034] running on GCE node
I20260808 04:02:39.885929 28673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:39.887475 28673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:39.888698 28673 hybrid_clock.cc:648] HybridClock initialized: now 1786161759888632 us; error 63 us; skew 500 ppm
I20260808 04:02:39.889060 28673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:39.891424 28673 webserver.cc:460] Webserver started at http://127.28.251.254:33823/ using document root <none> and password file <none>
I20260808 04:02:39.892154 28673 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:39.892270 28673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:39.892596 28673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:39.895661 28673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "5123bc83700d4a64919089f2811f9b52"
format_stamp: "Formatted at 2026-08-08 04:02:39 on dist-test-slave-pww0"
server_key: "bf7a2eeed626bbef6fbc47eaeff373b9"
server_key_iv: "fcc47edf8b22c10d261431af0f2e1f5f"
server_key_version: "encryptionkey@0"
I20260808 04:02:39.896523 28673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "5123bc83700d4a64919089f2811f9b52"
format_stamp: "Formatted at 2026-08-08 04:02:39 on dist-test-slave-pww0"
server_key: "bf7a2eeed626bbef6fbc47eaeff373b9"
server_key_iv: "fcc47edf8b22c10d261431af0f2e1f5f"
server_key_version: "encryptionkey@0"
I20260808 04:02:39.901901 28673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:02:39.905699 28697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:39.907227 28673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:02:39.907442 28673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "5123bc83700d4a64919089f2811f9b52"
format_stamp: "Formatted at 2026-08-08 04:02:39 on dist-test-slave-pww0"
server_key: "bf7a2eeed626bbef6fbc47eaeff373b9"
server_key_iv: "fcc47edf8b22c10d261431af0f2e1f5f"
server_key_version: "encryptionkey@0"
I20260808 04:02:39.907754 28673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:39.927399 28673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:39.928427 28673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:39.928743 28673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:39.947729 28673 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:40635
I20260808 04:02:39.947749 28786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:40635 every 8 connection(s)
I20260808 04:02:39.949146 28673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:02:39.950590 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 28673
I20260808 04:02:39.950779 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:02:39.951344 29679 external_mini_cluster.cc:1428] Setting key 955004c4fc0c91c545966dc0c5d95993
I20260808 04:02:39.954121 28787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:39.965974 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:39.953393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.28.251.254:40635)
0808 04:02:39.954463 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:39.954483 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:39.954528 (+ 45us) server_negotiation.cc:407] Connection header received
0808 04:02:39.956248 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:39.956271 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:39.956749 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:39.957107 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:39.957803 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:39.959157 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:39.961197 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:39.961899 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:39.962031 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:39.963403 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:39.963411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:39.963430 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:39.964128 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:39.964213 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:39.964239 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:39.964335 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:39.964591 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 04:02:39.964965 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":399,"threads_started":1}
I20260808 04:02:39.966861 28787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52: Bootstrap starting.
I20260808 04:02:39.970170 28787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:39.971289 28787 log.cc:824] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:39.974218 28787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52: No bootstrap required, opened a new log
I20260808 04:02:39.981815 28787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5123bc83700d4a64919089f2811f9b52" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40635 } }
I20260808 04:02:39.982237 28787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:39.982347 28787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5123bc83700d4a64919089f2811f9b52, State: Initialized, Role: FOLLOWER
I20260808 04:02:39.983016 28787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5123bc83700d4a64919089f2811f9b52" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40635 } }
I20260808 04:02:39.983256 28787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:02:39.983397 28787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:02:39.983593 28787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:39.985849 28787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5123bc83700d4a64919089f2811f9b52" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40635 } }
I20260808 04:02:39.986404 28787 leader_election.cc:302] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5123bc83700d4a64919089f2811f9b52; no voters:
I20260808 04:02:39.987048 28787 leader_election.cc:288] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:02:39.987226 28792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:39.988023 28792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [term 1 LEADER]: Becoming Leader. State: Replica: 5123bc83700d4a64919089f2811f9b52, State: Running, Role: LEADER
I20260808 04:02:39.988561 28792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5123bc83700d4a64919089f2811f9b52" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40635 } }
I20260808 04:02:39.989295 28787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:02:39.991498 28796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5123bc83700d4a64919089f2811f9b52. Latest consensus state: current_term: 1 leader_uuid: "5123bc83700d4a64919089f2811f9b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5123bc83700d4a64919089f2811f9b52" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40635 } } }
I20260808 04:02:39.991936 28796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:39.995651 28795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5123bc83700d4a64919089f2811f9b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5123bc83700d4a64919089f2811f9b52" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 40635 } } }
I20260808 04:02:39.995913 28795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:39.996573 28804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:02:40.002080 28804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:02:40.011234 28804 catalog_manager.cc:1347] Generated new cluster ID: a973e0ca850940c78cbe43b349957bd8
I20260808 04:02:40.011372 28804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:02:40.046849 28804 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:02:40.048416 28804 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:02:40.061524 28804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52: Generated new TSK 0
I20260808 04:02:40.062774 28804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:02:40.080438 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40635
--builtin_ntp_servers=127.28.251.212:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:02:40.450167 28821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:40.450498 28821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:40.450591 28821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:40.458226 28821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:40.458364 28821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:40.458479 28821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:02:40.468546 28821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:40.469897 28821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:40.471899 28821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:40.485881 28829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:40.485881 28830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:40.486455 28821 server_base.cc:1034] running on GCE node
W20260808 04:02:40.486491 28834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:40.487273 28821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:40.488404 28821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:40.489683 28821 hybrid_clock.cc:648] HybridClock initialized: now 1786161760489624 us; error 45 us; skew 500 ppm
I20260808 04:02:40.490084 28821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:40.492733 28821 webserver.cc:460] Webserver started at http://127.28.251.193:37973/ using document root <none> and password file <none>
I20260808 04:02:40.493501 28821 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:40.493614 28821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:40.493950 28821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:40.497383 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "9202213944cb4e7992a0f5e81eb2b7b5"
format_stamp: "Formatted at 2026-08-08 04:02:40 on dist-test-slave-pww0"
server_key: "f91778010d46ddd99b4fa330cc1842df"
server_key_iv: "d924af711a66f1b325892c15eb946811"
server_key_version: "encryptionkey@0"
I20260808 04:02:40.498246 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "9202213944cb4e7992a0f5e81eb2b7b5"
format_stamp: "Formatted at 2026-08-08 04:02:40 on dist-test-slave-pww0"
server_key: "f91778010d46ddd99b4fa330cc1842df"
server_key_iv: "d924af711a66f1b325892c15eb946811"
server_key_version: "encryptionkey@0"
I20260808 04:02:40.503993 28821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 04:02:40.508158 28844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:40.509649 28821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:02:40.509838 28821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "9202213944cb4e7992a0f5e81eb2b7b5"
format_stamp: "Formatted at 2026-08-08 04:02:40 on dist-test-slave-pww0"
server_key: "f91778010d46ddd99b4fa330cc1842df"
server_key_iv: "d924af711a66f1b325892c15eb946811"
server_key_version: "encryptionkey@0"
I20260808 04:02:40.510077 28821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:40.525247 28821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:40.526186 28821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:40.526525 28821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:40.527767 28821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:40.529431 28821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:40.529541 28821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:40.529664 28821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:40.529747 28821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:40.563060 28821 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:45451
I20260808 04:02:40.563099 29026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:45451 every 8 connection(s)
I20260808 04:02:40.564527 28821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:02:40.568239 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 28821
I20260808 04:02:40.568454 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 04:02:40.569072 29679 external_mini_cluster.cc:1428] Setting key d33d522b276cf7f3b165891ae63268f5
I20260808 04:02:40.572110 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40635
--builtin_ntp_servers=127.28.251.212:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:02:40.578372 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:40.567351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:56099 (local address 127.28.251.254:40635)
0808 04:02:40.567893 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:40.567898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:40.569456 (+ 1558us) server_negotiation.cc:407] Connection header received
0808 04:02:40.571398 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:40.571404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:40.571525 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:40.571629 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:40.572947 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:40.573696 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:40.575368 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:40.575860 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:40.575983 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:40.577452 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:40.577457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:40.577460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:40.577482 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:40.577527 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:40.577538 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:40.577614 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:40.577870 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 04:02:40.577986 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":290,"threads_started":1}
I20260808 04:02:40.579507 29039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:40.568052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40635 (local address 127.28.251.193:56099)
0808 04:02:40.569257 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:40.569313 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:40.571094 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:40.571930 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:40.571941 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:40.572044 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:40.572752 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:40.572780 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:40.573872 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:40.573876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:40.575202 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:40.575216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:40.576117 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:40.576126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:40.576368 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:40.577092 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:02:40.577094 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:40.577142 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:02:40.577235 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:40.577634 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:40.577647 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:40.577654 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:02:40.577839 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 04:02:40.578288 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":405,"threads_started":1}
I20260808 04:02:40.581427 29028 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40635
I20260808 04:02:40.581779 29028 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:40.582639 29028 heartbeater.cc:511] Master 127.28.251.254:40635 requested a full tablet report, sending...
I20260808 04:02:40.585028 28722 ts_manager.cc:194] Registered new tserver with Master: 9202213944cb4e7992a0f5e81eb2b7b5 (127.28.251.193:45451)
I20260808 04:02:40.586596 28722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.193:56099
W20260808 04:02:40.942597 29040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:40.942915 29040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:40.943033 29040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:40.950713 29040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:40.950851 29040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:40.950992 29040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:02:40.960892 29040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:40.962317 29040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:40.964555 29040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:40.978515 29053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:40.978596 29050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:40.978559 29051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:40.979833 29040 server_base.cc:1034] running on GCE node
I20260808 04:02:40.980518 29040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:40.981601 29040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:40.982832 29040 hybrid_clock.cc:648] HybridClock initialized: now 1786161760982784 us; error 38 us; skew 500 ppm
I20260808 04:02:40.983178 29040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:40.985816 29040 webserver.cc:460] Webserver started at http://127.28.251.194:35325/ using document root <none> and password file <none>
I20260808 04:02:40.986553 29040 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:40.986651 29040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:40.986951 29040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:40.990342 29040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "c9c746af6da142a0885bf5a57b38a98b"
format_stamp: "Formatted at 2026-08-08 04:02:40 on dist-test-slave-pww0"
server_key: "2b0d633d7682f2f823204ac29100f4a5"
server_key_iv: "3549e3c5e7e351402f2b9e53bb82f026"
server_key_version: "encryptionkey@0"
I20260808 04:02:40.991212 29040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "c9c746af6da142a0885bf5a57b38a98b"
format_stamp: "Formatted at 2026-08-08 04:02:40 on dist-test-slave-pww0"
server_key: "2b0d633d7682f2f823204ac29100f4a5"
server_key_iv: "3549e3c5e7e351402f2b9e53bb82f026"
server_key_version: "encryptionkey@0"
I20260808 04:02:40.997112 29040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260808 04:02:41.001377 29061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:41.002933 29040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:02:41.003115 29040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "c9c746af6da142a0885bf5a57b38a98b"
format_stamp: "Formatted at 2026-08-08 04:02:40 on dist-test-slave-pww0"
server_key: "2b0d633d7682f2f823204ac29100f4a5"
server_key_iv: "3549e3c5e7e351402f2b9e53bb82f026"
server_key_version: "encryptionkey@0"
I20260808 04:02:41.003393 29040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:41.032490 29040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:41.033453 29040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:41.033789 29040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:41.034874 29040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:41.036599 29040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:41.036783 29040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:41.036950 29040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:41.037030 29040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:41.070797 29040 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:38091
I20260808 04:02:41.070933 29263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:38091 every 8 connection(s)
I20260808 04:02:41.072230 29040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:02:41.081384 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 29040
I20260808 04:02:41.081573 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 04:02:41.082026 29679 external_mini_cluster.cc:1428] Setting key 012749175ca8d8d2090a60e8bb2ade8f
I20260808 04:02:41.085227 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40635
--builtin_ntp_servers=127.28.251.212:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:02:41.086040 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.074975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:49305 (local address 127.28.251.254:40635)
0808 04:02:41.075214 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:41.075219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:41.076813 (+ 1594us) server_negotiation.cc:407] Connection header received
0808 04:02:41.078661 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:41.078667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:41.078743 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:41.078844 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:41.080208 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.081010 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.082889 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.083419 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.083578 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.085207 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:41.085212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:41.085217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:41.085238 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:41.085273 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:41.085285 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:41.085375 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:41.085681 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 04:02:41.085796 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 04:02:41.087579 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.075566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40635 (local address 127.28.251.194:49305)
0808 04:02:41.076594 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:41.076639 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:41.078348 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:41.079124 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:41.079133 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:41.079232 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:41.079968 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.079998 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.081202 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.081206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.082689 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.082708 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.083691 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.083700 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.083988 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.084840 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:02:41.084842 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:41.084895 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:02:41.084987 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:41.085419 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:41.085433 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:41.085444 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 04:02:41.085652 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 04:02:41.086097 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":321,"threads_started":1}
I20260808 04:02:41.089131 29264 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40635
I20260808 04:02:41.089529 29264 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:41.090543 29264 heartbeater.cc:511] Master 127.28.251.254:40635 requested a full tablet report, sending...
I20260808 04:02:41.092243 28722 ts_manager.cc:194] Registered new tserver with Master: c9c746af6da142a0885bf5a57b38a98b (127.28.251.194:38091)
I20260808 04:02:41.093135 28722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.194:49305
W20260808 04:02:41.458443 29268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:41.458786 29268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:41.458884 29268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:41.466508 29268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:41.466636 29268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:41.466760 29268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:02:41.476368 29268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:40635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:41.477703 29268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:41.479609 29268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:41.494377 29288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:41.494458 29297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:41.494436 29291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:41.495203 29268 server_base.cc:1034] running on GCE node
I20260808 04:02:41.495967 29268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:41.497160 29268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:41.498443 29268 hybrid_clock.cc:648] HybridClock initialized: now 1786161761498384 us; error 52 us; skew 500 ppm
I20260808 04:02:41.498775 29268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:41.501492 29268 webserver.cc:460] Webserver started at http://127.28.251.195:43971/ using document root <none> and password file <none>
I20260808 04:02:41.502246 29268 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:41.502354 29268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:41.502660 29268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:41.505990 29268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "a960af7072b74827bc90955d848b8b76"
format_stamp: "Formatted at 2026-08-08 04:02:41 on dist-test-slave-pww0"
server_key: "f571c91bc58bd8dafa294eabcbd486fb"
server_key_iv: "e771aa66526af31c15599fc7cc4f5ef8"
server_key_version: "encryptionkey@0"
I20260808 04:02:41.506870 29268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "a960af7072b74827bc90955d848b8b76"
format_stamp: "Formatted at 2026-08-08 04:02:41 on dist-test-slave-pww0"
server_key: "f571c91bc58bd8dafa294eabcbd486fb"
server_key_iv: "e771aa66526af31c15599fc7cc4f5ef8"
server_key_version: "encryptionkey@0"
I20260808 04:02:41.513108 29268 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260808 04:02:41.517184 29306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:41.518785 29268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:02:41.518965 29268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "a960af7072b74827bc90955d848b8b76"
format_stamp: "Formatted at 2026-08-08 04:02:41 on dist-test-slave-pww0"
server_key: "f571c91bc58bd8dafa294eabcbd486fb"
server_key_iv: "e771aa66526af31c15599fc7cc4f5ef8"
server_key_version: "encryptionkey@0"
I20260808 04:02:41.519186 29268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:41.535600 29268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:41.536572 29268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:41.536909 29268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:41.538036 29268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:41.539690 29268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:41.539790 29268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:41.539908 29268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:41.539981 29268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:41.573432 29268 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:39675
I20260808 04:02:41.573477 29493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:39675 every 8 connection(s)
I20260808 04:02:41.574913 29268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:02:41.584141 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 29268
I20260808 04:02:41.584326 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 04:02:41.584774 29679 external_mini_cluster.cc:1428] Setting key df5be331efa1f2f0d0036481e1feacd1
I20260808 04:02:41.588465 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.577578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:56817 (local address 127.28.251.254:40635)
0808 04:02:41.577760 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:41.577765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:41.579254 (+ 1489us) server_negotiation.cc:407] Connection header received
0808 04:02:41.581246 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:41.581251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:41.581346 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:41.581445 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:41.582700 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.583531 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.585393 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.585906 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.586031 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.587578 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:41.587585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:41.587589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:41.587611 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:41.587650 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:41.587663 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:41.587736 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:41.588053 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 04:02:41.588154 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:02:41.589617 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.578070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:40635 (local address 127.28.251.195:56817)
0808 04:02:41.579058 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:41.579105 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:41.580916 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:41.581710 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:41.581720 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:41.581828 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:41.582506 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.582527 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.583713 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.583718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.585209 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.585226 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.586170 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.586178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.586432 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.587187 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:02:41.587190 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:41.587236 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:02:41.587345 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:41.587753 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:41.587767 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:41.587775 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:02:41.587980 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 04:02:41.588439 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":310,"threads_started":1}
I20260808 04:02:41.590410 29028 heartbeater.cc:503] Master 127.28.251.254:40635 was elected leader, sending a full tablet report...
I20260808 04:02:41.591323 29494 heartbeater.cc:347] Connected to a master server at 127.28.251.254:40635
I20260808 04:02:41.591689 29494 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:41.592607 29494 heartbeater.cc:511] Master 127.28.251.254:40635 requested a full tablet report, sending...
I20260808 04:02:41.594213 28722 ts_manager.cc:194] Registered new tserver with Master: a960af7072b74827bc90955d848b8b76 (127.28.251.195:39675)
I20260808 04:02:41.595033 28722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.195:56817
I20260808 04:02:41.600514 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:02:41.611826 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.604726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.28.251.254:40635)
0808 04:02:41.605025 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:41.605033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:41.605549 (+ 516us) server_negotiation.cc:407] Connection header received
0808 04:02:41.605698 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:41.605702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:41.605787 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:41.605886 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:41.606486 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.607262 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.608950 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.609467 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.609605 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.610863 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:41.610867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:41.610870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:41.610892 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:41.610929 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:41.610942 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:41.611060 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:41.611328 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 04:02:41.611401 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:02:41.616575 28722 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:02:41.624221 28722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:02:41.652163 29519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.638441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:45451 (local address 127.0.0.1:57240)
0808 04:02:41.641827 (+ 3386us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:41.641846 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:41.641979 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:41.643196 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:41.643206 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:41.643243 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:41.644482 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.644499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.646532 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.646537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.648859 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.648876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.651075 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.651082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.651292 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.651346 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 04:02:41.651499 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 04:02:41.651737 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3242,"mutex_wait_us":69,"thread_start_us":1843,"threads_started":1}
I20260808 04:02:41.652637 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.638422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.28.251.194:38091)
0808 04:02:41.644172 (+ 5750us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:41.644184 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:41.644222 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:02:41.644329 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:41.644338 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:41.644630 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:41.644859 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:41.645482 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.646960 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.649578 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.651014 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.651177 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.651389 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:41.652104 (+ 715us) server_negotiation.cc:299] Negotiation successful
0808 04:02:41.652302 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5563,"thread_start_us":5427,"threads_started":1}
I20260808 04:02:41.652661 29522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.638813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.28.251.193:45451)
0808 04:02:41.640098 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:41.640110 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:41.642142 (+ 2032us) server_negotiation.cc:407] Connection header received
0808 04:02:41.642273 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:41.642281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:41.642611 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:41.642849 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:41.644712 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.646348 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.649064 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.650461 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.650666 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.650923 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:41.652145 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0808 04:02:41.652339 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":941,"threads_started":1}
I20260808 04:02:41.652908 29521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.638194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:38091 (local address 127.0.0.1:47340)
0808 04:02:41.640823 (+ 2629us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:41.640874 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:41.641109 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:41.644859 (+ 3750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:41.644865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:41.644895 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:41.645256 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.645268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.647168 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.647175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.649348 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.649365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.651551 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.651558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.651746 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.651777 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:02:41.651860 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 04:02:41.651980 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2303,"mutex_wait_us":49,"thread_start_us":3288,"threads_started":1}
I20260808 04:02:41.653371 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.642875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:39675 (local address 127.0.0.1:38256)
0808 04:02:41.644627 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:41.644646 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:41.644787 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:41.645473 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:41.645477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:41.645507 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:41.645943 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.645958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.648168 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.648175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.650087 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.650109 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.652545 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.652555 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.652755 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.652794 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 04:02:41.652883 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 04:02:41.653009 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1638,"mutex_wait_us":4,"thread_start_us":782,"threads_started":1}
I20260808 04:02:41.653682 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.642361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.28.251.195:39675)
0808 04:02:41.644286 (+ 1925us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:41.644293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:41.644755 (+ 462us) server_negotiation.cc:407] Connection header received
0808 04:02:41.644995 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:41.645003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:41.645231 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:41.645424 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:41.646418 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.647999 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.650318 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.651812 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.652030 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.652247 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:41.653063 (+ 816us) server_negotiation.cc:299] Negotiation successful
0808 04:02:41.653264 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1756,"thread_start_us":1582,"threads_started":1}
I20260808 04:02:41.657485 29384 tablet_service.cc:1467] Processing CreateTablet for tablet 047f71dd82b943a6820b5b9d74822d67 (DEFAULT_TABLE table=test-table [id=fb74e0536b394590b91771a6299d95ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:41.657481 28926 tablet_service.cc:1467] Processing CreateTablet for tablet 047f71dd82b943a6820b5b9d74822d67 (DEFAULT_TABLE table=test-table [id=fb74e0536b394590b91771a6299d95ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:41.657481 29149 tablet_service.cc:1467] Processing CreateTablet for tablet 047f71dd82b943a6820b5b9d74822d67 (DEFAULT_TABLE table=test-table [id=fb74e0536b394590b91771a6299d95ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:41.658916 29384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047f71dd82b943a6820b5b9d74822d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:41.658929 28926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047f71dd82b943a6820b5b9d74822d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:41.658953 29149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047f71dd82b943a6820b5b9d74822d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:41.673146 29526 tablet_bootstrap.cc:492] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76: Bootstrap starting.
I20260808 04:02:41.675494 29527 tablet_bootstrap.cc:492] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b: Bootstrap starting.
I20260808 04:02:41.678023 29526 tablet_bootstrap.cc:654] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:41.679088 29528 tablet_bootstrap.cc:492] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5: Bootstrap starting.
I20260808 04:02:41.679636 29526 log.cc:824] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:41.679720 29527 tablet_bootstrap.cc:654] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:41.681093 29527 log.cc:824] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:41.682734 29528 tablet_bootstrap.cc:654] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:41.684126 29528 log.cc:824] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:41.686293 29527 tablet_bootstrap.cc:492] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b: No bootstrap required, opened a new log
I20260808 04:02:41.686650 29527 ts_tablet_manager.cc:1397] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.011s
I20260808 04:02:41.686813 29528 tablet_bootstrap.cc:492] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5: No bootstrap required, opened a new log
I20260808 04:02:41.687119 29528 ts_tablet_manager.cc:1397] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260808 04:02:41.689006 29526 tablet_bootstrap.cc:492] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76: No bootstrap required, opened a new log
I20260808 04:02:41.689317 29526 ts_tablet_manager.cc:1397] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20260808 04:02:41.695745 29527 raft_consensus.cc:348] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c746af6da142a0885bf5a57b38a98b" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38091 } } peers { permanent_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45451 } } peers { permanent_uuid: "a960af7072b74827bc90955d848b8b76" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 39675 } }
I20260808 04:02:41.696254 29527 raft_consensus.cc:374] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:41.696378 29527 raft_consensus.cc:729] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c746af6da142a0885bf5a57b38a98b, State: Initialized, Role: FOLLOWER
I20260808 04:02:41.697127 29527 consensus_queue.cc:260] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c746af6da142a0885bf5a57b38a98b" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38091 } } peers { permanent_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45451 } } peers { permanent_uuid: "a960af7072b74827bc90955d848b8b76" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 39675 } }
I20260808 04:02:41.698756 29264 heartbeater.cc:503] Master 127.28.251.254:40635 was elected leader, sending a full tablet report...
I20260808 04:02:41.698935 29528 raft_consensus.cc:348] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c746af6da142a0885bf5a57b38a98b" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38091 } } peers { permanent_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45451 } } peers { permanent_uuid: "a960af7072b74827bc90955d848b8b76" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 39675 } }
I20260808 04:02:41.699416 29528 raft_consensus.cc:374] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:41.699579 29528 raft_consensus.cc:729] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9202213944cb4e7992a0f5e81eb2b7b5, State: Initialized, Role: FOLLOWER
I20260808 04:02:41.699615 29527 ts_tablet_manager.cc:1428] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260808 04:02:41.700279 29528 consensus_queue.cc:260] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c746af6da142a0885bf5a57b38a98b" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38091 } } peers { permanent_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45451 } } peers { permanent_uuid: "a960af7072b74827bc90955d848b8b76" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 39675 } }
I20260808 04:02:41.700224 29526 raft_consensus.cc:348] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c746af6da142a0885bf5a57b38a98b" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38091 } } peers { permanent_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45451 } } peers { permanent_uuid: "a960af7072b74827bc90955d848b8b76" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 39675 } }
I20260808 04:02:41.700692 29526 raft_consensus.cc:374] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:41.700796 29526 raft_consensus.cc:729] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a960af7072b74827bc90955d848b8b76, State: Initialized, Role: FOLLOWER
I20260808 04:02:41.701439 29526 consensus_queue.cc:260] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c746af6da142a0885bf5a57b38a98b" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38091 } } peers { permanent_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45451 } } peers { permanent_uuid: "a960af7072b74827bc90955d848b8b76" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 39675 } }
I20260808 04:02:41.702500 29528 ts_tablet_manager.cc:1428] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260808 04:02:41.703727 29494 heartbeater.cc:503] Master 127.28.251.254:40635 was elected leader, sending a full tablet report...
I20260808 04:02:41.704449 29526 ts_tablet_manager.cc:1428] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260808 04:02:41.711673 29539 raft_consensus.cc:482] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:41.712044 29539 raft_consensus.cc:504] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c746af6da142a0885bf5a57b38a98b" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38091 } } peers { permanent_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45451 } } peers { permanent_uuid: "a960af7072b74827bc90955d848b8b76" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 39675 } }
I20260808 04:02:41.713996 29539 leader_election.cc:288] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9c746af6da142a0885bf5a57b38a98b (127.28.251.194:38091), a960af7072b74827bc90955d848b8b76 (127.28.251.195:39675)
I20260808 04:02:41.720433 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.714413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:47067 (local address 127.28.251.194:38091)
0808 04:02:41.714677 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:41.714683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:41.714971 (+ 288us) server_negotiation.cc:407] Connection header received
0808 04:02:41.715216 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:41.715223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:41.715323 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:41.715416 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:41.716080 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.716791 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.718606 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.719511 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.719658 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.719733 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:41.720017 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 04:02:41.720156 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:02:41.720520 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.714201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:38091 (local address 127.28.251.193:47067)
0808 04:02:41.714859 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:41.714875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:41.715044 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:41.715576 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:41.715580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:41.715607 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:41.715928 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.715937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.716922 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.716926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.718405 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.718418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.719708 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.719714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.719864 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.719889 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:02:41.719973 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:02:41.720117 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":21,"thread_start_us":318,"threads_started":1}
I20260808 04:02:41.721107 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.715083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:59309 (local address 127.28.251.195:39675)
0808 04:02:41.715270 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:41.715277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:41.715778 (+ 501us) server_negotiation.cc:407] Connection header received
0808 04:02:41.715936 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:41.715942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:41.716009 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:41.716083 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:41.716619 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.717339 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.719161 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.720043 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.720202 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.720275 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:41.720733 (+ 458us) server_negotiation.cc:299] Negotiation successful
0808 04:02:41.720876 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:02:41.721794 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.714943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:39675 (local address 127.28.251.193:59309)
0808 04:02:41.715672 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:41.715686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:41.715816 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:41.716110 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:41.716114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:41.716138 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:41.716467 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.716476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.717484 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.717490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.718976 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:41.718993 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.720239 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.720246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:41.720424 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.720449 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:02:41.721323 (+ 874us) client_negotiation.cc:240] Negotiation successful
0808 04:02:41.721473 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":1,"thread_start_us":356,"threads_started":1}
I20260808 04:02:41.721717 29183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047f71dd82b943a6820b5b9d74822d67" candidate_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9c746af6da142a0885bf5a57b38a98b" is_pre_election: true
I20260808 04:02:41.722374 29183 raft_consensus.cc:2454] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9202213944cb4e7992a0f5e81eb2b7b5 in term 0.
I20260808 04:02:41.722925 29420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047f71dd82b943a6820b5b9d74822d67" candidate_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a960af7072b74827bc90955d848b8b76" is_pre_election: true
I20260808 04:02:41.723222 28847 leader_election.cc:302] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9202213944cb4e7992a0f5e81eb2b7b5, c9c746af6da142a0885bf5a57b38a98b; no voters:
I20260808 04:02:41.723634 29420 raft_consensus.cc:2454] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9202213944cb4e7992a0f5e81eb2b7b5 in term 0.
I20260808 04:02:41.723722 29539 raft_consensus.cc:2781] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:02:41.723866 29539 raft_consensus.cc:482] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:02:41.723991 29539 raft_consensus.cc:3037] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:41.726460 29539 raft_consensus.cc:504] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c746af6da142a0885bf5a57b38a98b" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38091 } } peers { permanent_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45451 } } peers { permanent_uuid: "a960af7072b74827bc90955d848b8b76" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 39675 } }
I20260808 04:02:41.727110 29539 leader_election.cc:288] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [CANDIDATE]: Term 1 election: Requested vote from peers c9c746af6da142a0885bf5a57b38a98b (127.28.251.194:38091), a960af7072b74827bc90955d848b8b76 (127.28.251.195:39675)
I20260808 04:02:41.727756 29183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047f71dd82b943a6820b5b9d74822d67" candidate_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9c746af6da142a0885bf5a57b38a98b"
I20260808 04:02:41.727995 29183 raft_consensus.cc:3037] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:41.728103 29420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047f71dd82b943a6820b5b9d74822d67" candidate_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a960af7072b74827bc90955d848b8b76"
I20260808 04:02:41.728338 29420 raft_consensus.cc:3037] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:41.730469 29183 raft_consensus.cc:2454] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9202213944cb4e7992a0f5e81eb2b7b5 in term 1.
I20260808 04:02:41.730690 29420 raft_consensus.cc:2454] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9202213944cb4e7992a0f5e81eb2b7b5 in term 1.
I20260808 04:02:41.730965 28847 leader_election.cc:302] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9202213944cb4e7992a0f5e81eb2b7b5, c9c746af6da142a0885bf5a57b38a98b; no voters:
I20260808 04:02:41.731279 29539 raft_consensus.cc:2781] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:41.731814 29539 raft_consensus.cc:686] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9202213944cb4e7992a0f5e81eb2b7b5, State: Running, Role: LEADER
I20260808 04:02:41.732399 29539 consensus_queue.cc:237] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c746af6da142a0885bf5a57b38a98b" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38091 } } peers { permanent_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45451 } } peers { permanent_uuid: "a960af7072b74827bc90955d848b8b76" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 39675 } }
I20260808 04:02:41.737882 28718 catalog_manager.cc:5495] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9202213944cb4e7992a0f5e81eb2b7b5 (127.28.251.193). New cstate: current_term: 1 leader_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c746af6da142a0885bf5a57b38a98b" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9202213944cb4e7992a0f5e81eb2b7b5" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 45451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a960af7072b74827bc90955d848b8b76" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 39675 } health_report { overall_health: UNKNOWN } } }
I20260808 04:02:41.755564 29522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.749731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.28.251.193:45451)
0808 04:02:41.749989 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:41.749994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:41.750018 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:02:41.750259 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:41.750264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:41.750385 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:41.750537 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:02:41.750884 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.751693 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.753210 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.753686 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.753815 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.754337 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:02:41.754910 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:02:41.755055 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:41.755179 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 04:02:41.755239 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 04:02:41.764513 29183 raft_consensus.cc:1261] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b [term 1 FOLLOWER]: Refusing update from remote peer 9202213944cb4e7992a0f5e81eb2b7b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:41.764513 29420 raft_consensus.cc:1261] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76 [term 1 FOLLOWER]: Refusing update from remote peer 9202213944cb4e7992a0f5e81eb2b7b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:41.765515 29543 consensus_queue.cc:1035] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a960af7072b74827bc90955d848b8b76" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:02:41.765894 29539 consensus_queue.cc:1035] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9c746af6da142a0885bf5a57b38a98b" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:02:41.781428 29548 mvcc.cc:205] Tried to move back new op lower bound from 7316118576177000448 to 7316118576060309504. Current Snapshot: MvccSnapshot[applied={T|T < 7316118576177000448 or (T in {7316118576177000448})}]
I20260808 04:02:41.792825 28718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:41.793111 28718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:41.797257 28718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5123bc83700d4a64919089f2811f9b52: Sending DeleteTablet for 3 replicas of tablet 047f71dd82b943a6820b5b9d74822d67
I20260808 04:02:41.798678 28926 tablet_service.cc:1514] Processing DeleteTablet for tablet 047f71dd82b943a6820b5b9d74822d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:41 UTC) from {username='slave'} at 127.0.0.1:57240
I20260808 04:02:41.798722 29149 tablet_service.cc:1514] Processing DeleteTablet for tablet 047f71dd82b943a6820b5b9d74822d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:41 UTC) from {username='slave'} at 127.0.0.1:47340
I20260808 04:02:41.798722 29384 tablet_service.cc:1514] Processing DeleteTablet for tablet 047f71dd82b943a6820b5b9d74822d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:41 UTC) from {username='slave'} at 127.0.0.1:38256
I20260808 04:02:41.804630 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 28821
I20260808 04:02:41.810227 29563 tablet_replica.cc:330] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5: stopping tablet replica
I20260808 04:02:41.810853 29563 raft_consensus.cc:2229] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:02:41.812083 29559 tablet_replica.cc:330] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76: stopping tablet replica
I20260808 04:02:41.812585 29563 raft_consensus.cc:2258] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:41.812857 29559 raft_consensus.cc:2229] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:41.813484 29559 raft_consensus.cc:2258] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:41.814461 29560 tablet_replica.cc:330] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b: stopping tablet replica
I20260808 04:02:41.815896 29563 ts_tablet_manager.cc:1905] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:41.816794 29559 ts_tablet_manager.cc:1905] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:41.817402 29560 raft_consensus.cc:2229] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:41.825022 29563 ts_tablet_manager.cc:1918] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:41.825273 29563 log.cc:1196] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/047f71dd82b943a6820b5b9d74822d67
I20260808 04:02:41.825939 29563 ts_tablet_manager.cc:1939] T 047f71dd82b943a6820b5b9d74822d67 P 9202213944cb4e7992a0f5e81eb2b7b5: Deleting consensus metadata
I20260808 04:02:41.825953 29560 raft_consensus.cc:2258] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:41.826773 29559 ts_tablet_manager.cc:1918] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:41.828078 28698 catalog_manager.cc:4841] TS 9202213944cb4e7992a0f5e81eb2b7b5 (127.28.251.193:45451): tablet 047f71dd82b943a6820b5b9d74822d67 (table test-table [id=fb74e0536b394590b91771a6299d95ae]) successfully deleted
I20260808 04:02:41.828619 29559 log.cc:1196] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/047f71dd82b943a6820b5b9d74822d67
I20260808 04:02:41.829270 29560 ts_tablet_manager.cc:1905] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:41.829456 29559 ts_tablet_manager.cc:1939] T 047f71dd82b943a6820b5b9d74822d67 P a960af7072b74827bc90955d848b8b76: Deleting consensus metadata
I20260808 04:02:41.829710 29522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.815686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.28.251.193:45451)
0808 04:02:41.816045 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:41.816052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:41.820162 (+ 4110us) server_negotiation.cc:407] Connection header received
0808 04:02:41.820745 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:41.820752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:41.820890 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:41.820973 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:41.821772 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.823084 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.825126 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.825782 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.825893 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.827528 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:41.827541 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:41.827561 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:41.828389 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:41.828457 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:41.828488 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:41.828583 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:41.829107 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 04:02:41.829229 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260808 04:02:41.830632 29017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:41.831678 28700 catalog_manager.cc:4841] TS a960af7072b74827bc90955d848b8b76 (127.28.251.195:39675): tablet 047f71dd82b943a6820b5b9d74822d67 (table test-table [id=fb74e0536b394590b91771a6299d95ae]) successfully deleted
I20260808 04:02:41.837582 29560 ts_tablet_manager.cc:1918] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:41.837826 29560 log.cc:1196] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/047f71dd82b943a6820b5b9d74822d67
I20260808 04:02:41.838455 29560 ts_tablet_manager.cc:1939] T 047f71dd82b943a6820b5b9d74822d67 P c9c746af6da142a0885bf5a57b38a98b: Deleting consensus metadata
I20260808 04:02:41.840603 28700 catalog_manager.cc:4841] TS c9c746af6da142a0885bf5a57b38a98b (127.28.251.194:38091): tablet 047f71dd82b943a6820b5b9d74822d67 (table test-table [id=fb74e0536b394590b91771a6299d95ae]) successfully deleted
I20260808 04:02:41.933240 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 28821
I20260808 04:02:41.961265 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 29040
I20260808 04:02:41.968444 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:41.962055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.28.251.194:38091)
0808 04:02:41.962237 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:41.962242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:41.962265 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:02:41.962486 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:41.962490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:41.962599 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:41.962686 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:41.963119 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.963924 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.965257 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:41.965692 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:41.965812 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:41.966930 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:41.966940 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:41.966957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:41.967646 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:41.967708 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:41.967731 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:41.967848 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:41.967982 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 04:02:41.968109 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:02:41.969183 29258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:42.067269 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 29040
I20260808 04:02:42.095156 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 29268
I20260808 04:02:42.102049 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:42.096031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.28.251.195:39675)
0808 04:02:42.096201 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:42.096206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:42.096241 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:02:42.096425 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:42.096429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:42.096546 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:42.096644 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:42.097050 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:42.097910 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:42.099240 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:42.099732 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:42.099836 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:42.100835 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:42.100847 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:42.100866 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:42.101461 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:42.101513 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:42.101535 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:42.101637 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:42.101702 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 04:02:42.101798 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:02:42.102761 29483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:42.194377 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 29268
I20260808 04:02:42.223821 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 28673
I20260808 04:02:42.224967 28780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:42.335775 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 28673
2026-08-08T04:02:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2963 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:02:42 dist-test-slave-pww0 krb5kdc[29573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:02:42 dist-test-slave-pww0 krb5kdc[29573](info): set up 2 sockets
Aug 08 04:02:42 dist-test-slave-pww0 krb5kdc[29573](info): commencing operation
krb5kdc: starting...
W20260808 04:02:43.650368 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.004s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:02:43 dist-test-slave-pww0 krb5kdc[29573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:02:44.016835 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:36403
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:36403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:44.386112 29595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:44.386449 29595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:44.386559 29595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:44.394174 29595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:02:44.394306 29595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:44.394357 29595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:44.394423 29595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:02:44.394469 29595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:02:44.404872 29595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:36403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:36403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:44.406302 29595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:44.407733 29595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:44.420051 29609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:44.420006 29611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:44.420085 29613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:44.421128 29595 server_base.cc:1034] running on GCE node
I20260808 04:02:44.421967 29595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:44.423573 29595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:44.424841 29595 hybrid_clock.cc:648] HybridClock initialized: now 1786161764424767 us; error 48 us; skew 500 ppm
I20260808 04:02:44.425246 29595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:44 dist-test-slave-pww0 krb5kdc[29573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161764, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:44.432281 29595 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:02:44.434042 29595 webserver.cc:460] Webserver started at http://127.28.251.254:46827/ using document root <none> and password file <none>
I20260808 04:02:44.434823 29595 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:44.434954 29595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:44.435272 29595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:44.438592 29595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "603f3b27324f43c8a46b052d7b1d2960"
format_stamp: "Formatted at 2026-08-08 04:02:44 on dist-test-slave-pww0"
server_key: "0971d85392e696cc992cdbdd8a890402"
server_key_iv: "0ba6d5df3aa9a496bd65434f2f04eaa7"
server_key_version: "encryptionkey@0"
I20260808 04:02:44.439380 29595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "603f3b27324f43c8a46b052d7b1d2960"
format_stamp: "Formatted at 2026-08-08 04:02:44 on dist-test-slave-pww0"
server_key: "0971d85392e696cc992cdbdd8a890402"
server_key_iv: "0ba6d5df3aa9a496bd65434f2f04eaa7"
server_key_version: "encryptionkey@0"
I20260808 04:02:44.444846 29595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:02:44.448510 29628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:44.449942 29595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:02:44.450122 29595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "603f3b27324f43c8a46b052d7b1d2960"
format_stamp: "Formatted at 2026-08-08 04:02:44 on dist-test-slave-pww0"
server_key: "0971d85392e696cc992cdbdd8a890402"
server_key_iv: "0ba6d5df3aa9a496bd65434f2f04eaa7"
server_key_version: "encryptionkey@0"
I20260808 04:02:44.450320 29595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:44.472882 29595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:44.477270 29595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:44.477571 29595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:44.496838 29595 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:36403
I20260808 04:02:44.496861 29726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:36403 every 8 connection(s)
I20260808 04:02:44.498337 29595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:02:44.502396 29732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:44.504688 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 29595
I20260808 04:02:44.504897 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:02:44.505452 29679 external_mini_cluster.cc:1428] Setting key 235bf279b8ccbce6b306f1f7a0a32e28
I20260808 04:02:44.515417 29732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960: Bootstrap starting.
I20260808 04:02:44.519658 29732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960: Neither blocks nor log segments found. Creating new log.
Aug 08 04:02:44 dist-test-slave-pww0 krb5kdc[29573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:44.520977 29732 log.cc:824] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:44.524497 29732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960: No bootstrap required, opened a new log
I20260808 04:02:44.530607 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:44.507500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.28.251.254:36403)
0808 04:02:44.508361 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:44.508374 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:44.508428 (+ 54us) server_negotiation.cc:407] Connection header received
0808 04:02:44.510184 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:44.510208 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:44.510645 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:44.510988 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:44.512900 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:44.514269 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:44.516160 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:44.516897 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:44.517063 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:44.522329 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:44.522343 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:44.522366 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:44.522416 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:44.525671 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:44.526716 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:44.526724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:44.526732 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:44.526955 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:44.527441 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:44.527444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:44.527446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:44.528118 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:44.528411 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:44.528910 (+ 499us) server_negotiation.cc:299] Negotiation successful
0808 04:02:44.529347 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":392,"threads_started":1}
I20260808 04:02:44.533919 29732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f3b27324f43c8a46b052d7b1d2960" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36403 } }
I20260808 04:02:44.534336 29732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:44.534449 29732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603f3b27324f43c8a46b052d7b1d2960, State: Initialized, Role: FOLLOWER
I20260808 04:02:44.535061 29732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f3b27324f43c8a46b052d7b1d2960" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36403 } }
I20260808 04:02:44.535297 29732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:02:44.535434 29732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:02:44.535648 29732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:44.537992 29732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f3b27324f43c8a46b052d7b1d2960" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36403 } }
I20260808 04:02:44.538607 29732 leader_election.cc:302] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 603f3b27324f43c8a46b052d7b1d2960; no voters:
I20260808 04:02:44.539269 29732 leader_election.cc:288] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:02:44.539496 29738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:44.540652 29738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [term 1 LEADER]: Becoming Leader. State: Replica: 603f3b27324f43c8a46b052d7b1d2960, State: Running, Role: LEADER
I20260808 04:02:44.541281 29738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f3b27324f43c8a46b052d7b1d2960" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36403 } }
I20260808 04:02:44.541718 29732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:02:44.544238 29742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 603f3b27324f43c8a46b052d7b1d2960. Latest consensus state: current_term: 1 leader_uuid: "603f3b27324f43c8a46b052d7b1d2960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f3b27324f43c8a46b052d7b1d2960" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36403 } } }
I20260808 04:02:44.544742 29742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:44.545200 29741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "603f3b27324f43c8a46b052d7b1d2960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f3b27324f43c8a46b052d7b1d2960" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36403 } } }
I20260808 04:02:44.545874 29741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:44.546501 29752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:02:44.552457 29752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:02:44.563171 29752 catalog_manager.cc:1347] Generated new cluster ID: deacc4f64d524e40b53dc7675470423a
I20260808 04:02:44.563326 29752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:02:44.581625 29752 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:02:44.583664 29752 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:02:44.595206 29752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960: Generated new TSK 0
I20260808 04:02:44.596405 29752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
Entry for principal kudu/127.28.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab.
I20260808 04:02:44.688531 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36403
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:45.062772 29770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:45.063094 29770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:45.063223 29770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:45.070866 29770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:45.071002 29770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:45.071151 29770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:02:45.081434 29770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.193.keytab
--principal=kudu/127.28.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:45.082883 29770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:45.084303 29770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:45.097918 29782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:45.097949 29781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:45.098269 29785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:45.098863 29770 server_base.cc:1034] running on GCE node
I20260808 04:02:45.099556 29770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:45.100594 29770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:45.101804 29770 hybrid_clock.cc:648] HybridClock initialized: now 1786161765101732 us; error 66 us; skew 500 ppm
I20260808 04:02:45.102131 29770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:45 dist-test-slave-pww0 krb5kdc[29573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161765, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:45.105372 29770 init.cc:377] Logged in from keytab as kudu/127.28.251.193@KRBTEST.COM (short username kudu)
I20260808 04:02:45.106900 29770 webserver.cc:460] Webserver started at http://127.28.251.193:35287/ using document root <none> and password file <none>
I20260808 04:02:45.107669 29770 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:45.107802 29770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:45.108129 29770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:45.111214 29770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "19f7b6a158f44671841c69bec913c1b3"
format_stamp: "Formatted at 2026-08-08 04:02:45 on dist-test-slave-pww0"
server_key: "3b82763e2e6b727363b77373f6429346"
server_key_iv: "c36ae69b46549a4fdbea46cf6e43388c"
server_key_version: "encryptionkey@0"
I20260808 04:02:45.112071 29770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "19f7b6a158f44671841c69bec913c1b3"
format_stamp: "Formatted at 2026-08-08 04:02:45 on dist-test-slave-pww0"
server_key: "3b82763e2e6b727363b77373f6429346"
server_key_iv: "c36ae69b46549a4fdbea46cf6e43388c"
server_key_version: "encryptionkey@0"
I20260808 04:02:45.117569 29770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:02:45.121546 29795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:45.123003 29770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:02:45.123301 29770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "19f7b6a158f44671841c69bec913c1b3"
format_stamp: "Formatted at 2026-08-08 04:02:45 on dist-test-slave-pww0"
server_key: "3b82763e2e6b727363b77373f6429346"
server_key_iv: "c36ae69b46549a4fdbea46cf6e43388c"
server_key_version: "encryptionkey@0"
I20260808 04:02:45.123584 29770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:45.146622 29770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:45.150946 29770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:45.151244 29770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:45.152351 29770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:45.153827 29770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:45.153918 29770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:45.154042 29770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:45.154098 29770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:45.186425 29770 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:46455
I20260808 04:02:45.186503 30008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:46455 every 8 connection(s)
I20260808 04:02:45.187885 29770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:02:45.196195 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 29770
I20260808 04:02:45.196381 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 04:02:45.196848 29679 external_mini_cluster.cc:1428] Setting key 11a85c1404415859499d5959dc68b96c
Aug 08 04:02:45 dist-test-slave-pww0 krb5kdc[29573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161765, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.193@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:45.209054 30011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:45.190656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:47285 (local address 127.28.251.254:36403)
0808 04:02:45.191146 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:45.191153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:45.192190 (+ 1037us) server_negotiation.cc:407] Connection header received
0808 04:02:45.194037 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:45.194042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:45.194118 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:45.194218 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:45.195877 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:45.196676 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:45.198730 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:45.199175 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:45.199295 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:45.203070 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:45.203075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:45.203079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:45.203117 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:45.205737 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:45.206565 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:45.206570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:45.206572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:45.206700 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:45.207167 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:45.207171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:45.207173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:45.207483 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:45.207627 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:45.208419 (+ 792us) server_negotiation.cc:299] Negotiation successful
0808 04:02:45.208672 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":223,"threads_started":1}
I20260808 04:02:45.210114 30012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:45.191240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:36403 (local address 127.28.251.193:47285)
0808 04:02:45.192024 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:45.192063 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:45.193757 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:45.194452 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:45.194461 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:45.195017 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:45.195677 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:45.195699 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:45.196850 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:45.196856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:45.198573 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:45.198588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:45.199440 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:45.199449 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:45.199715 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:45.200392 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:02:45.200423 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:45.202798 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:45.205915 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:45.205926 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:45.205946 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:45.206425 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:45.206812 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:45.206817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:45.206822 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:45.207040 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:45.207627 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:45.207636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:45.208011 (+ 375us) client_negotiation.cc:769] Sending connection context
0808 04:02:45.208269 (+ 258us) client_negotiation.cc:240] Negotiation successful
0808 04:02:45.208858 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":308,"threads_started":1}
I20260808 04:02:45.211952 30009 heartbeater.cc:347] Connected to a master server at 127.28.251.254:36403
I20260808 04:02:45.212311 30009 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:45.213191 30009 heartbeater.cc:511] Master 127.28.251.254:36403 requested a full tablet report, sending...
I20260808 04:02:45.216008 29660 ts_manager.cc:194] Registered new tserver with Master: 19f7b6a158f44671841c69bec913c1b3 (127.28.251.193:46455)
I20260808 04:02:45.217586 29660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.193@KRBTEST.COM'} at 127.28.251.193:47285
WARNING: no policy specified for kudu/127.28.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
Entry for principal kudu/127.28.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab.
I20260808 04:02:45.277001 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36403
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:45.645066 30017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:45.645404 30017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:45.645573 30017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:45.653330 30017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:45.653469 30017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:45.653638 30017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:02:45.663976 30017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.194.keytab
--principal=kudu/127.28.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:45.665375 30017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:45.666806 30017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:45.681082 30026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:45.681068 30028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:45.681320 30025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:45.681982 30017 server_base.cc:1034] running on GCE node
I20260808 04:02:45.682637 30017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:45.683660 30017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:45.684885 30017 hybrid_clock.cc:648] HybridClock initialized: now 1786161765684832 us; error 40 us; skew 500 ppm
I20260808 04:02:45.685218 30017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:45 dist-test-slave-pww0 krb5kdc[29573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161765, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:45.688635 30017 init.cc:377] Logged in from keytab as kudu/127.28.251.194@KRBTEST.COM (short username kudu)
I20260808 04:02:45.690340 30017 webserver.cc:460] Webserver started at http://127.28.251.194:46091/ using document root <none> and password file <none>
I20260808 04:02:45.691093 30017 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:45.691223 30017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:45.691627 30017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:45.694689 30017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "fa7c123240824ec99b6268f7d3665699"
format_stamp: "Formatted at 2026-08-08 04:02:45 on dist-test-slave-pww0"
server_key: "b6ed1d0863a6db5d3a7eba83e71f9dd0"
server_key_iv: "8ad1835edb61793e9c8073523a3c161d"
server_key_version: "encryptionkey@0"
I20260808 04:02:45.695514 30017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "fa7c123240824ec99b6268f7d3665699"
format_stamp: "Formatted at 2026-08-08 04:02:45 on dist-test-slave-pww0"
server_key: "b6ed1d0863a6db5d3a7eba83e71f9dd0"
server_key_iv: "8ad1835edb61793e9c8073523a3c161d"
server_key_version: "encryptionkey@0"
I20260808 04:02:45.700994 30017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 04:02:45.704591 30041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:45.705930 30017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:02:45.706096 30017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "fa7c123240824ec99b6268f7d3665699"
format_stamp: "Formatted at 2026-08-08 04:02:45 on dist-test-slave-pww0"
server_key: "b6ed1d0863a6db5d3a7eba83e71f9dd0"
server_key_iv: "8ad1835edb61793e9c8073523a3c161d"
server_key_version: "encryptionkey@0"
I20260808 04:02:45.706303 30017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:45.738555 30017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:45.743047 30017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:45.743331 30017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:45.744357 30017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:45.745784 30017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:45.745859 30017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:45.745977 30017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:45.746026 30017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:45.776700 30017 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:35749
I20260808 04:02:45.776750 30234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:35749 every 8 connection(s)
I20260808 04:02:45.778024 30017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:02:45.783888 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 30017
I20260808 04:02:45.784082 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 04:02:45.784564 29679 external_mini_cluster.cc:1428] Setting key 9cc73722498cf177105490a9cd35b7fa
Aug 08 04:02:45 dist-test-slave-pww0 krb5kdc[29573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161765, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.194@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:45.800668 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:45.780701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:52707 (local address 127.28.251.254:36403)
0808 04:02:45.781318 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:45.781323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:45.782081 (+ 758us) server_negotiation.cc:407] Connection header received
0808 04:02:45.784062 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:45.784068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:45.784164 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:45.784279 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:45.786175 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:45.787016 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:45.789552 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:45.790037 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:45.790166 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:45.794752 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:45.794758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:45.794762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:45.794821 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:45.797368 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:45.798296 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:45.798300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:45.798302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:45.798434 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:45.798900 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:45.798903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:45.798905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:45.799173 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:45.799287 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:45.800190 (+ 903us) server_negotiation.cc:299] Negotiation successful
0808 04:02:45.800373 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":254,"threads_started":1}
I20260808 04:02:45.802094 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:45.781169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:36403 (local address 127.28.251.194:52707)
0808 04:02:45.781905 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:45.781958 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:45.783750 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:45.784566 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:45.784575 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:45.785173 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:45.785958 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:45.785990 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:45.787207 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:45.787213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:45.789310 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:45.789333 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:45.790342 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:45.790362 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:45.790673 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:45.791529 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:02:45.791561 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:45.794403 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:45.797540 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:45.797551 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:45.797572 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:45.798170 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:45.798537 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:45.798542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:45.798545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:45.798793 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:45.799289 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:45.799297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:45.799792 (+ 495us) client_negotiation.cc:769] Sending connection context
0808 04:02:45.800110 (+ 318us) client_negotiation.cc:240] Negotiation successful
0808 04:02:45.800663 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":288,"threads_started":1}
I20260808 04:02:45.803586 30236 heartbeater.cc:347] Connected to a master server at 127.28.251.254:36403
I20260808 04:02:45.803934 30236 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:45.804675 30236 heartbeater.cc:511] Master 127.28.251.254:36403 requested a full tablet report, sending...
I20260808 04:02:45.806305 29660 ts_manager.cc:194] Registered new tserver with Master: fa7c123240824ec99b6268f7d3665699 (127.28.251.194:35749)
I20260808 04:02:45.807504 29660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.194@KRBTEST.COM'} at 127.28.251.194:52707
WARNING: no policy specified for kudu/127.28.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
Entry for principal kudu/127.28.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab.
I20260808 04:02:45.863068 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36403
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:02:46.221643 30009 heartbeater.cc:503] Master 127.28.251.254:36403 was elected leader, sending a full tablet report...
W20260808 04:02:46.230451 30245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:46.230767 30245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:46.230872 30245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:46.238873 30245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:46.238994 30245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:46.239121 30245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:02:46.249625 30245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.195.keytab
--principal=kudu/127.28.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:36403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:46.251085 30245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:46.252521 30245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:46.266443 30251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:46.266474 30260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:46.266508 30254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:46.267570 30245 server_base.cc:1034] running on GCE node
I20260808 04:02:46.268261 30245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:46.269344 30245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:46.270603 30245 hybrid_clock.cc:648] HybridClock initialized: now 1786161766270539 us; error 43 us; skew 500 ppm
I20260808 04:02:46.270954 30245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:46 dist-test-slave-pww0 krb5kdc[29573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161766, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:46.274206 30245 init.cc:377] Logged in from keytab as kudu/127.28.251.195@KRBTEST.COM (short username kudu)
I20260808 04:02:46.275956 30245 webserver.cc:460] Webserver started at http://127.28.251.195:39725/ using document root <none> and password file <none>
I20260808 04:02:46.276707 30245 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:46.276827 30245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:46.277149 30245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:46.280340 30245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "885853806c2848db99cc42588c0d9edf"
format_stamp: "Formatted at 2026-08-08 04:02:46 on dist-test-slave-pww0"
server_key: "4eb1b9753bd3d7854f3c6cc6a9501899"
server_key_iv: "8dd909e8b9068b40849d846be635c03a"
server_key_version: "encryptionkey@0"
I20260808 04:02:46.281112 30245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "885853806c2848db99cc42588c0d9edf"
format_stamp: "Formatted at 2026-08-08 04:02:46 on dist-test-slave-pww0"
server_key: "4eb1b9753bd3d7854f3c6cc6a9501899"
server_key_iv: "8dd909e8b9068b40849d846be635c03a"
server_key_version: "encryptionkey@0"
I20260808 04:02:46.286535 30245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 04:02:46.290208 30273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:46.291724 30245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:02:46.291900 30245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "885853806c2848db99cc42588c0d9edf"
format_stamp: "Formatted at 2026-08-08 04:02:46 on dist-test-slave-pww0"
server_key: "4eb1b9753bd3d7854f3c6cc6a9501899"
server_key_iv: "8dd909e8b9068b40849d846be635c03a"
server_key_version: "encryptionkey@0"
I20260808 04:02:46.292114 30245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:46.311506 30245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:46.315980 30245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:46.316339 30245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:46.317476 30245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:46.318974 30245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:46.319057 30245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:46.319195 30245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:46.319252 30245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:46.352056 30245 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:36487
I20260808 04:02:46.352131 30451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:36487 every 8 connection(s)
I20260808 04:02:46.353526 30245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:02:46.361382 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 30245
I20260808 04:02:46.361573 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 04:02:46.362010 29679 external_mini_cluster.cc:1428] Setting key 649b935f11f9fdaf651646ec837a32b3
Aug 08 04:02:46 dist-test-slave-pww0 krb5kdc[29573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161766, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.195@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:02:46.374393 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.356287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:50425 (local address 127.28.251.254:36403)
0808 04:02:46.356768 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:46.356775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:46.357837 (+ 1062us) server_negotiation.cc:407] Connection header received
0808 04:02:46.359766 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:46.359771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:46.359850 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:46.359943 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:46.361649 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.362479 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.364185 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.364606 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.364724 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.368284 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:46.368289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:46.368292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:46.368337 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:46.371029 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:46.371913 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:46.371918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:46.371920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:46.372056 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:46.372512 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:46.372516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:46.372518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:46.372807 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:46.372950 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:46.373723 (+ 773us) server_negotiation.cc:299] Negotiation successful
0808 04:02:46.373972 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":214,"threads_started":1}
I20260808 04:02:46.375384 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.356923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:36403 (local address 127.28.251.195:50425)
0808 04:02:46.357661 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:46.357703 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:46.359441 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:46.360237 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:46.360246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:46.360843 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:46.361446 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.361469 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.362654 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.362657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.364033 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.364047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.364851 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.364860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.365085 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.365740 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:02:46.365768 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:46.368061 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:46.371205 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:46.371216 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:46.371231 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:46.371772 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:46.372175 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:02:46.372179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:02:46.372181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:02:46.372391 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:02:46.372945 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:46.372953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:46.373325 (+ 372us) client_negotiation.cc:769] Sending connection context
0808 04:02:46.373583 (+ 258us) client_negotiation.cc:240] Negotiation successful
0808 04:02:46.374181 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":275,"threads_started":1}
I20260808 04:02:46.377034 30453 heartbeater.cc:347] Connected to a master server at 127.28.251.254:36403
I20260808 04:02:46.377394 30453 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:46.378180 30453 heartbeater.cc:511] Master 127.28.251.254:36403 requested a full tablet report, sending...
I20260808 04:02:46.379647 29660 ts_manager.cc:194] Registered new tserver with Master: 885853806c2848db99cc42588c0d9edf (127.28.251.195:36487)
I20260808 04:02:46.380470 29660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.251.195@KRBTEST.COM'} at 127.28.251.195:50425
I20260808 04:02:46.389930 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:02:46.407423 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.394537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.28.251.254:36403)
0808 04:02:46.394763 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:46.394770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:46.395073 (+ 303us) server_negotiation.cc:407] Connection header received
0808 04:02:46.395253 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:46.395258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:46.395346 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:46.395436 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:46.397075 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.397813 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.399111 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.399576 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.399689 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.402151 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:46.402156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:46.402159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:46.402197 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:46.404610 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:46.405327 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:46.405333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:46.405337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:46.405479 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:46.405860 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:46.405866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:46.405868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:46.406211 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:46.406449 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:46.406851 (+ 402us) server_negotiation.cc:299] Negotiation successful
0808 04:02:46.407089 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:02:46.411549 29660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:02:46.418432 29660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:02:46.441962 30476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.431206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:36487 (local address 127.0.0.1:45918)
0808 04:02:46.432347 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:46.432392 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:46.432593 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:46.433978 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:46.433986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:46.434020 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:46.434418 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.434432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.436722 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.436731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.439180 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.439193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.440941 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.440947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.441135 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.441193 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 04:02:46.441345 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 04:02:46.441562 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":664,"threads_started":1}
I20260808 04:02:46.442013 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.431522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.28.251.195:36487)
0808 04:02:46.433282 (+ 1760us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:46.433291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:46.433330 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:02:46.433434 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:46.433442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:46.433698 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:46.434644 (+ 946us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:46.434953 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.436595 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.439416 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.440724 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.440994 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.441185 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:46.441451 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 04:02:46.441640 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1584,"thread_start_us":1425,"threads_started":1}
I20260808 04:02:46.442896 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.432862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.28.251.194:35749)
0808 04:02:46.433358 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:46.433366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:46.435108 (+ 1742us) server_negotiation.cc:407] Connection header received
0808 04:02:46.435275 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:46.435283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:46.435558 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:46.435752 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:46.436317 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.437599 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.440024 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.441428 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.441587 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.441741 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:46.442371 (+ 630us) server_negotiation.cc:299] Negotiation successful
0808 04:02:46.442546 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":309,"threads_started":1}
I20260808 04:02:46.443637 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.431626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:35749 (local address 127.0.0.1:48434)
0808 04:02:46.434956 (+ 3330us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:46.434985 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:46.435113 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:46.435752 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:46.435757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:46.435784 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:46.436147 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.436157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.437734 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.437739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.439841 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.439860 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.441906 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.441912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.442095 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.442127 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:02:46.443104 (+ 977us) client_negotiation.cc:240] Negotiation successful
0808 04:02:46.443264 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3206,"thread_start_us":3711,"threads_started":1}
I20260808 04:02:46.444119 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.432426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46455 (local address 127.0.0.1:52482)
0808 04:02:46.435638 (+ 3212us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:46.435655 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:46.435771 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:46.436389 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:46.436393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:46.436422 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:46.436818 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.436836 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.438734 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.438740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.440792 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.440806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.443366 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.443371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.443525 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.443557 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:02:46.443649 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:02:46.443774 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3097,"thread_start_us":2242,"threads_started":1}
I20260808 04:02:46.444170 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.433509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.28.251.193:46455)
0808 04:02:46.434258 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:46.434267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:46.435896 (+ 1629us) server_negotiation.cc:407] Connection header received
0808 04:02:46.435997 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:46.436005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:46.436215 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:46.436396 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:46.437013 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.438584 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.440988 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.442836 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.443042 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.443210 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:46.443711 (+ 501us) server_negotiation.cc:299] Negotiation successful
0808 04:02:46.443865 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":399,"threads_started":1}
I20260808 04:02:46.446679 30355 tablet_service.cc:1467] Processing CreateTablet for tablet 0af9ebacc4324df591dcd6331df66be4 (DEFAULT_TABLE table=test-table [id=dba19a4b4d4e4adb93fe91e38ff930b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:46.446955 30129 tablet_service.cc:1467] Processing CreateTablet for tablet 0af9ebacc4324df591dcd6331df66be4 (DEFAULT_TABLE table=test-table [id=dba19a4b4d4e4adb93fe91e38ff930b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:46.448144 30129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af9ebacc4324df591dcd6331df66be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:46.450305 29888 tablet_service.cc:1467] Processing CreateTablet for tablet 0af9ebacc4324df591dcd6331df66be4 (DEFAULT_TABLE table=test-table [id=dba19a4b4d4e4adb93fe91e38ff930b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:46.448180 30355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af9ebacc4324df591dcd6331df66be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:46.451550 29888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af9ebacc4324df591dcd6331df66be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:46.465372 30483 tablet_bootstrap.cc:492] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3: Bootstrap starting.
I20260808 04:02:46.466709 30487 tablet_bootstrap.cc:492] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: Bootstrap starting.
I20260808 04:02:46.467752 30485 tablet_bootstrap.cc:492] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: Bootstrap starting.
I20260808 04:02:46.469250 30483 tablet_bootstrap.cc:654] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:46.470685 30483 log.cc:824] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:46.470832 30485 tablet_bootstrap.cc:654] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:46.470988 30487 tablet_bootstrap.cc:654] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:46.472184 30485 log.cc:824] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:46.472833 30487 log.cc:824] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:46.474776 30485 tablet_bootstrap.cc:492] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: No bootstrap required, opened a new log
I20260808 04:02:46.474777 30483 tablet_bootstrap.cc:492] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3: No bootstrap required, opened a new log
I20260808 04:02:46.475052 30485 ts_tablet_manager.cc:1397] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 04:02:46.475075 30483 ts_tablet_manager.cc:1397] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260808 04:02:46.478516 30487 tablet_bootstrap.cc:492] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: No bootstrap required, opened a new log
I20260808 04:02:46.478826 30487 ts_tablet_manager.cc:1397] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260808 04:02:46.483181 30485 raft_consensus.cc:348] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885853806c2848db99cc42588c0d9edf" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36487 } } peers { permanent_uuid: "fa7c123240824ec99b6268f7d3665699" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 35749 } } peers { permanent_uuid: "19f7b6a158f44671841c69bec913c1b3" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46455 } }
I20260808 04:02:46.483871 30485 raft_consensus.cc:374] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:46.483987 30485 raft_consensus.cc:729] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7c123240824ec99b6268f7d3665699, State: Initialized, Role: FOLLOWER
I20260808 04:02:46.484637 30485 consensus_queue.cc:260] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885853806c2848db99cc42588c0d9edf" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36487 } } peers { permanent_uuid: "fa7c123240824ec99b6268f7d3665699" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 35749 } } peers { permanent_uuid: "19f7b6a158f44671841c69bec913c1b3" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46455 } }
I20260808 04:02:46.485868 30236 heartbeater.cc:503] Master 127.28.251.254:36403 was elected leader, sending a full tablet report...
I20260808 04:02:46.485580 30483 raft_consensus.cc:348] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885853806c2848db99cc42588c0d9edf" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36487 } } peers { permanent_uuid: "fa7c123240824ec99b6268f7d3665699" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 35749 } } peers { permanent_uuid: "19f7b6a158f44671841c69bec913c1b3" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46455 } }
I20260808 04:02:46.486151 30483 raft_consensus.cc:374] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:46.486269 30483 raft_consensus.cc:729] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f7b6a158f44671841c69bec913c1b3, State: Initialized, Role: FOLLOWER
I20260808 04:02:46.486546 30485 ts_tablet_manager.cc:1428] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260808 04:02:46.486949 30483 consensus_queue.cc:260] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885853806c2848db99cc42588c0d9edf" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36487 } } peers { permanent_uuid: "fa7c123240824ec99b6268f7d3665699" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 35749 } } peers { permanent_uuid: "19f7b6a158f44671841c69bec913c1b3" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46455 } }
I20260808 04:02:46.488900 30483 ts_tablet_manager.cc:1428] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260808 04:02:46.491006 30487 raft_consensus.cc:348] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885853806c2848db99cc42588c0d9edf" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36487 } } peers { permanent_uuid: "fa7c123240824ec99b6268f7d3665699" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 35749 } } peers { permanent_uuid: "19f7b6a158f44671841c69bec913c1b3" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46455 } }
I20260808 04:02:46.491611 30487 raft_consensus.cc:374] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:46.491756 30487 raft_consensus.cc:729] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885853806c2848db99cc42588c0d9edf, State: Initialized, Role: FOLLOWER
I20260808 04:02:46.492506 30487 consensus_queue.cc:260] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885853806c2848db99cc42588c0d9edf" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36487 } } peers { permanent_uuid: "fa7c123240824ec99b6268f7d3665699" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 35749 } } peers { permanent_uuid: "19f7b6a158f44671841c69bec913c1b3" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46455 } }
I20260808 04:02:46.494206 30453 heartbeater.cc:503] Master 127.28.251.254:36403 was elected leader, sending a full tablet report...
I20260808 04:02:46.495051 30487 ts_tablet_manager.cc:1428] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
W20260808 04:02:46.530037 30240 tablet.cc:2369] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:02:46.561959 30494 raft_consensus.cc:482] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:46.562291 30494 raft_consensus.cc:504] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885853806c2848db99cc42588c0d9edf" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36487 } } peers { permanent_uuid: "fa7c123240824ec99b6268f7d3665699" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 35749 } } peers { permanent_uuid: "19f7b6a158f44671841c69bec913c1b3" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46455 } }
I20260808 04:02:46.564150 30494 leader_election.cc:288] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 885853806c2848db99cc42588c0d9edf (127.28.251.195:36487), 19f7b6a158f44671841c69bec913c1b3 (127.28.251.193:46455)
I20260808 04:02:46.573161 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.564650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:40777 (local address 127.28.251.195:36487)
0808 04:02:46.564873 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:46.564880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:46.565671 (+ 791us) server_negotiation.cc:407] Connection header received
0808 04:02:46.565890 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:46.565894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:46.565957 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:46.566035 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:46.566892 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.568000 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.570792 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.572073 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.572228 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.572305 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:46.572731 (+ 426us) server_negotiation.cc:299] Negotiation successful
0808 04:02:46.572880 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:02:46.573158 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.564764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:43437 (local address 127.28.251.193:46455)
0808 04:02:46.564989 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:46.564996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:46.565753 (+ 757us) server_negotiation.cc:407] Connection header received
0808 04:02:46.565951 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:46.565955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:46.566016 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:46.566091 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:46.566787 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.567961 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.570792 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.572056 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.572228 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.572305 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:46.572737 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 04:02:46.572885 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:02:46.573350 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.564692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46455 (local address 127.28.251.194:43437)
0808 04:02:46.565630 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:46.565650 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:46.565813 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:46.566129 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:46.566133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:46.566156 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:46.566618 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.566633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.568115 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.568121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.570620 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.570636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.572302 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.572313 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.572494 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.572556 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 04:02:46.572650 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:02:46.572836 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":60,"thread_start_us":498,"threads_started":1}
I20260808 04:02:46.573376 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.564650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:36487 (local address 127.28.251.194:40777)
0808 04:02:46.565538 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:46.565559 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:46.565702 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:46.566078 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:46.566082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:46.566107 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:46.566667 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.566682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.568115 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.568121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.570620 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:46.570637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.572331 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.572337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:46.572503 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.572547 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 04:02:46.572650 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 04:02:46.572835 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":181,"thread_start_us":423,"threads_started":1}
I20260808 04:02:46.574558 29923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9ebacc4324df591dcd6331df66be4" candidate_uuid: "fa7c123240824ec99b6268f7d3665699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f7b6a158f44671841c69bec913c1b3" is_pre_election: true
I20260808 04:02:46.574556 30382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9ebacc4324df591dcd6331df66be4" candidate_uuid: "fa7c123240824ec99b6268f7d3665699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885853806c2848db99cc42588c0d9edf" is_pre_election: true
I20260808 04:02:46.575057 30382 raft_consensus.cc:2454] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7c123240824ec99b6268f7d3665699 in term 0.
I20260808 04:02:46.575085 29923 raft_consensus.cc:2454] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7c123240824ec99b6268f7d3665699 in term 0.
I20260808 04:02:46.575773 30045 leader_election.cc:302] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 885853806c2848db99cc42588c0d9edf, fa7c123240824ec99b6268f7d3665699; no voters:
I20260808 04:02:46.576200 30494 raft_consensus.cc:2781] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:02:46.576350 30494 raft_consensus.cc:482] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:02:46.576450 30494 raft_consensus.cc:3037] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:46.578650 30494 raft_consensus.cc:504] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885853806c2848db99cc42588c0d9edf" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36487 } } peers { permanent_uuid: "fa7c123240824ec99b6268f7d3665699" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 35749 } } peers { permanent_uuid: "19f7b6a158f44671841c69bec913c1b3" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46455 } }
I20260808 04:02:46.579267 30494 leader_election.cc:288] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [CANDIDATE]: Term 1 election: Requested vote from peers 885853806c2848db99cc42588c0d9edf (127.28.251.195:36487), 19f7b6a158f44671841c69bec913c1b3 (127.28.251.193:46455)
I20260808 04:02:46.579818 30382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9ebacc4324df591dcd6331df66be4" candidate_uuid: "fa7c123240824ec99b6268f7d3665699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885853806c2848db99cc42588c0d9edf"
I20260808 04:02:46.579836 29923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9ebacc4324df591dcd6331df66be4" candidate_uuid: "fa7c123240824ec99b6268f7d3665699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f7b6a158f44671841c69bec913c1b3"
I20260808 04:02:46.580026 29923 raft_consensus.cc:3037] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:46.580020 30382 raft_consensus.cc:3037] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:46.582250 29923 raft_consensus.cc:2454] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7c123240824ec99b6268f7d3665699 in term 1.
I20260808 04:02:46.582316 30382 raft_consensus.cc:2454] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7c123240824ec99b6268f7d3665699 in term 1.
I20260808 04:02:46.582686 30044 leader_election.cc:302] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f7b6a158f44671841c69bec913c1b3, fa7c123240824ec99b6268f7d3665699; no voters:
I20260808 04:02:46.582998 30494 raft_consensus.cc:2781] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:46.583441 30494 raft_consensus.cc:686] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 1 LEADER]: Becoming Leader. State: Replica: fa7c123240824ec99b6268f7d3665699, State: Running, Role: LEADER
I20260808 04:02:46.583946 30494 consensus_queue.cc:237] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885853806c2848db99cc42588c0d9edf" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36487 } } peers { permanent_uuid: "fa7c123240824ec99b6268f7d3665699" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 35749 } } peers { permanent_uuid: "19f7b6a158f44671841c69bec913c1b3" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46455 } }
I20260808 04:02:46.588855 29660 catalog_manager.cc:5495] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa7c123240824ec99b6268f7d3665699 (127.28.251.194). New cstate: current_term: 1 leader_uuid: "fa7c123240824ec99b6268f7d3665699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "885853806c2848db99cc42588c0d9edf" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa7c123240824ec99b6268f7d3665699" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 35749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19f7b6a158f44671841c69bec913c1b3" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46455 } health_report { overall_health: UNKNOWN } } }
W20260808 04:02:46.605239 30454 tablet.cc:2369] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:02:46.609167 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.603379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.28.251.194:35749)
0808 04:02:46.603632 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:46.603637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:46.603694 (+ 57us) server_negotiation.cc:407] Connection header received
0808 04:02:46.603894 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:46.603898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:46.603982 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:46.604105 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:02:46.604442 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.605178 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.606867 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.607290 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.607385 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.608019 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:02:46.608540 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:02:46.608660 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:46.608789 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 04:02:46.608882 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260808 04:02:46.617758 29923 raft_consensus.cc:1261] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3 [term 1 FOLLOWER]: Refusing update from remote peer fa7c123240824ec99b6268f7d3665699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:46.617756 30382 raft_consensus.cc:1261] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf [term 1 FOLLOWER]: Refusing update from remote peer fa7c123240824ec99b6268f7d3665699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:46.618767 30494 consensus_queue.cc:1035] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "885853806c2848db99cc42588c0d9edf" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 36487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:02:46.619078 30500 consensus_queue.cc:1035] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f7b6a158f44671841c69bec913c1b3" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:02:46.632946 30511 mvcc.cc:205] Tried to move back new op lower bound from 7316118596055855104 to 7316118595931385856. Current Snapshot: MvccSnapshot[applied={T|T < 7316118596055855104}]
I20260808 04:02:46.650444 29660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:46.650727 29660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:46.654891 29660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 603f3b27324f43c8a46b052d7b1d2960: Sending DeleteTablet for 3 replicas of tablet 0af9ebacc4324df591dcd6331df66be4
I20260808 04:02:46.656338 29888 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af9ebacc4324df591dcd6331df66be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:46 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:52482
I20260808 04:02:46.656301 30129 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af9ebacc4324df591dcd6331df66be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:46 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:48434
I20260808 04:02:46.656302 30355 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af9ebacc4324df591dcd6331df66be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:46 UTC) from {username='kudu', principal='kudu/127.28.251.254@KRBTEST.COM'} at 127.0.0.1:45918
I20260808 04:02:46.657558 30526 tablet_replica.cc:330] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: stopping tablet replica
I20260808 04:02:46.658110 30526 raft_consensus.cc:2229] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:02:46.658995 30526 raft_consensus.cc:2258] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:46.659943 30527 tablet_replica.cc:330] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3: stopping tablet replica
I20260808 04:02:46.660440 30527 raft_consensus.cc:2229] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:46.660665 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 29770
I20260808 04:02:46.661003 30527 raft_consensus.cc:2258] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:46.661865 30525 tablet_replica.cc:330] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: stopping tablet replica
I20260808 04:02:46.662269 30525 raft_consensus.cc:2229] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:46.662779 30525 raft_consensus.cc:2258] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:46.663340 30526 ts_tablet_manager.cc:1905] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:46.664819 30525 ts_tablet_manager.cc:1905] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:46.671025 30526 ts_tablet_manager.cc:1918] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:46.671370 30526 log.cc:1196] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/0af9ebacc4324df591dcd6331df66be4
I20260808 04:02:46.671628 30525 ts_tablet_manager.cc:1918] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:46.671913 30525 log.cc:1196] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/0af9ebacc4324df591dcd6331df66be4
I20260808 04:02:46.671908 30527 ts_tablet_manager.cc:1905] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:46.672063 30526 ts_tablet_manager.cc:1939] T 0af9ebacc4324df591dcd6331df66be4 P fa7c123240824ec99b6268f7d3665699: Deleting consensus metadata
I20260808 04:02:46.672696 30525 ts_tablet_manager.cc:1939] T 0af9ebacc4324df591dcd6331df66be4 P 885853806c2848db99cc42588c0d9edf: Deleting consensus metadata
I20260808 04:02:46.674177 29629 catalog_manager.cc:4841] TS fa7c123240824ec99b6268f7d3665699 (127.28.251.194:35749): tablet 0af9ebacc4324df591dcd6331df66be4 (table test-table [id=dba19a4b4d4e4adb93fe91e38ff930b8]) successfully deleted
I20260808 04:02:46.674724 29631 catalog_manager.cc:4841] TS 885853806c2848db99cc42588c0d9edf (127.28.251.195:36487): tablet 0af9ebacc4324df591dcd6331df66be4 (table test-table [id=dba19a4b4d4e4adb93fe91e38ff930b8]) successfully deleted
I20260808 04:02:46.679147 30527 ts_tablet_manager.cc:1918] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:46.679368 30527 log.cc:1196] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786161430375636-29679-0/minicluster-data/ts-0/wal/wals/0af9ebacc4324df591dcd6331df66be4
I20260808 04:02:46.680009 30527 ts_tablet_manager.cc:1939] T 0af9ebacc4324df591dcd6331df66be4 P 19f7b6a158f44671841c69bec913c1b3: Deleting consensus metadata
I20260808 04:02:46.681970 29630 catalog_manager.cc:4841] TS 19f7b6a158f44671841c69bec913c1b3 (127.28.251.193:46455): tablet 0af9ebacc4324df591dcd6331df66be4 (table test-table [id=dba19a4b4d4e4adb93fe91e38ff930b8]) successfully deleted
Aug 08 04:02:46 dist-test-slave-pww0 krb5kdc[29573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.193@KRBTEST.COM
I20260808 04:02:46.694044 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.662589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.28.251.193:46455)
0808 04:02:46.662767 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:46.662774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:46.676026 (+ 13252us) server_negotiation.cc:407] Connection header received
0808 04:02:46.676169 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:46.676176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:46.676300 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:46.676423 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:46.678157 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.679280 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.681217 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.681914 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.682050 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.686583 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:46.686605 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:46.686626 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:46.686678 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:46.690708 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:46.691617 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:46.691629 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:46.691633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:46.691819 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:46.692247 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:46.692253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:46.692256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:46.692770 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:46.693108 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:46.693388 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 04:02:46.693645 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:02:46.694937 29997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:46.785805 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 29770
I20260808 04:02:46.814512 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 30017
Aug 08 04:02:46 dist-test-slave-pww0 krb5kdc[29573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.194@KRBTEST.COM
I20260808 04:02:46.833132 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.815539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.28.251.194:35749)
0808 04:02:46.815748 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:46.815754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:46.815774 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:02:46.815833 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:46.815836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:46.815899 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:46.815995 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:46.817386 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.818202 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.819606 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.820191 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.820336 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.826129 (+ 5793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:46.826152 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:46.826170 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:46.826226 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:46.829520 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:46.830669 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:46.830679 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:46.830681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:46.830878 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:46.831344 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:46.831348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:46.831350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:46.831948 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:46.832259 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:46.832597 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 04:02:46.832814 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:02:46.833907 30227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:46.953138 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 30017
I20260808 04:02:46.979199 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 30245
Aug 08 04:02:46 dist-test-slave-pww0 krb5kdc[29573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.195@KRBTEST.COM
I20260808 04:02:46.995054 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:46.980133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.28.251.195:36487)
0808 04:02:46.980353 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:46.980358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:46.980380 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:02:46.980478 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:46.980482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:46.980550 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:46.980629 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:46.981966 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.982724 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.984058 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:46.984489 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:46.984586 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:46.989038 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:46.989056 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:46.989074 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:46.989123 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:46.992050 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:46.992848 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:46.992856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:46.992859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:46.993028 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:46.993423 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:46.993426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:46.993428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:46.993838 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:46.994088 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:46.994482 (+ 394us) server_negotiation.cc:299] Negotiation successful
0808 04:02:46.994688 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 04:02:46.996009 30442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:47.091984 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 30245
I20260808 04:02:47.119886 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 29595
I20260808 04:02:47.121063 29720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:47.236303 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 29595
2026-08-08T04:02:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4899 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T04:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:02:47Z Disabled control of system clock
I20260808 04:02:47.347146 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41453
--webserver_interface=127.28.251.254
--webserver_port=0
--builtin_ntp_servers=127.28.251.212:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:41453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:02:47.710839 30546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:47.711159 30546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:47.711256 30546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:47.718719 30546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:02:47.718849 30546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:47.718887 30546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:47.718941 30546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:02:47.718986 30546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:02:47.728761 30546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:41453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:47.730165 30546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:47.732245 30546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:47.744494 30555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:47.744560 30557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:47.744530 30559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:47.745361 30546 server_base.cc:1034] running on GCE node
I20260808 04:02:47.746196 30546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:47.747835 30546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:47.749051 30546 hybrid_clock.cc:648] HybridClock initialized: now 1786161767748999 us; error 42 us; skew 500 ppm
I20260808 04:02:47.749428 30546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:47.751857 30546 webserver.cc:460] Webserver started at http://127.28.251.254:44007/ using document root <none> and password file <none>
I20260808 04:02:47.752581 30546 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:47.752698 30546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:47.753033 30546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:47.756233 30546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "bda789cf018a410bbfa7410abbfe47c4"
format_stamp: "Formatted at 2026-08-08 04:02:47 on dist-test-slave-pww0"
server_key: "6872ac6ac52941c43946462685b9404c"
server_key_iv: "0a411c1d6d648804566ce145b82cdca2"
server_key_version: "encryptionkey@0"
I20260808 04:02:47.757176 30546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "bda789cf018a410bbfa7410abbfe47c4"
format_stamp: "Formatted at 2026-08-08 04:02:47 on dist-test-slave-pww0"
server_key: "6872ac6ac52941c43946462685b9404c"
server_key_iv: "0a411c1d6d648804566ce145b82cdca2"
server_key_version: "encryptionkey@0"
I20260808 04:02:47.762769 30546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:02:47.766592 30572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:47.768164 30546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:02:47.768365 30546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "bda789cf018a410bbfa7410abbfe47c4"
format_stamp: "Formatted at 2026-08-08 04:02:47 on dist-test-slave-pww0"
server_key: "6872ac6ac52941c43946462685b9404c"
server_key_iv: "0a411c1d6d648804566ce145b82cdca2"
server_key_version: "encryptionkey@0"
I20260808 04:02:47.768595 30546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:47.799027 30546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:47.800015 30546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:47.800305 30546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:47.818377 30546 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:41453
I20260808 04:02:47.818395 30657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:41453 every 8 connection(s)
I20260808 04:02:47.819864 30546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:02:47.823933 30658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:47.823969 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 30546
I20260808 04:02:47.824131 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:02:47.824707 29679 external_mini_cluster.cc:1428] Setting key 42588640ef036bee136c6c0caf936a66
I20260808 04:02:47.840216 30658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4: Bootstrap starting.
I20260808 04:02:47.840353 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:47.826709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.28.251.254:41453)
0808 04:02:47.827914 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:47.827930 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:47.827983 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:02:47.829644 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:47.829675 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:47.830336 (+ 661us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:47.830840 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:47.831438 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:47.833516 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:47.834872 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:47.835936 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:47.836132 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:47.837248 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:47.837262 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:47.837287 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:47.838081 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:47.838195 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:47.838224 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:47.838335 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:47.838616 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 04:02:47.838984 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":361,"threads_started":1}
I20260808 04:02:47.843592 30658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:47.844561 30658 log.cc:824] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:47.847393 30658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4: No bootstrap required, opened a new log
I20260808 04:02:47.854727 30658 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda789cf018a410bbfa7410abbfe47c4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41453 } }
I20260808 04:02:47.855123 30658 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:47.855242 30658 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda789cf018a410bbfa7410abbfe47c4, State: Initialized, Role: FOLLOWER
I20260808 04:02:47.855911 30658 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda789cf018a410bbfa7410abbfe47c4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41453 } }
I20260808 04:02:47.856148 30658 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:02:47.856294 30658 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:02:47.856460 30658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:47.858748 30658 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda789cf018a410bbfa7410abbfe47c4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41453 } }
I20260808 04:02:47.859311 30658 leader_election.cc:302] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda789cf018a410bbfa7410abbfe47c4; no voters:
I20260808 04:02:47.859987 30658 leader_election.cc:288] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:02:47.860129 30668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:47.860863 30668 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [term 1 LEADER]: Becoming Leader. State: Replica: bda789cf018a410bbfa7410abbfe47c4, State: Running, Role: LEADER
I20260808 04:02:47.861375 30668 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda789cf018a410bbfa7410abbfe47c4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41453 } }
I20260808 04:02:47.862151 30658 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:02:47.864427 30672 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda789cf018a410bbfa7410abbfe47c4. Latest consensus state: current_term: 1 leader_uuid: "bda789cf018a410bbfa7410abbfe47c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda789cf018a410bbfa7410abbfe47c4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41453 } } }
I20260808 04:02:47.864876 30672 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:47.864369 30669 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda789cf018a410bbfa7410abbfe47c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda789cf018a410bbfa7410abbfe47c4" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41453 } } }
I20260808 04:02:47.865573 30669 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:47.867184 30682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:02:47.872995 30682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:02:47.882886 30682 catalog_manager.cc:1347] Generated new cluster ID: dc122e46809c4a079d03cefde27866f3
I20260808 04:02:47.883010 30682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:02:47.903023 30682 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:02:47.904485 30682 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:02:47.920979 30682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4: Generated new TSK 0
I20260808 04:02:47.921926 30682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:02:47.927794 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.193:0
--local_ip_for_outbound_sockets=127.28.251.193
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41453
--builtin_ntp_servers=127.28.251.212:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:02:48.297938 30701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:48.298271 30701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:48.298393 30701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:48.305927 30701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:48.306044 30701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:48.306187 30701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.193
I20260808 04:02:48.315824 30701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.251.193
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.193
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:48.317319 30701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:48.319252 30701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:48.333712 30709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:48.333750 30708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:48.333725 30711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:48.334537 30701 server_base.cc:1034] running on GCE node
I20260808 04:02:48.335215 30701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:48.336360 30701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:48.337572 30701 hybrid_clock.cc:648] HybridClock initialized: now 1786161768337501 us; error 66 us; skew 500 ppm
I20260808 04:02:48.337919 30701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:48.340942 30701 webserver.cc:460] Webserver started at http://127.28.251.193:42111/ using document root <none> and password file <none>
I20260808 04:02:48.341745 30701 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:48.341856 30701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:48.342144 30701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:48.345304 30701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/data/instance:
uuid: "64ab99f9f9b448d793e63c180fbe2af1"
format_stamp: "Formatted at 2026-08-08 04:02:48 on dist-test-slave-pww0"
server_key: "6c7fc329fb7cebfff79b01a5dac354fc"
server_key_iv: "a9d868339564815dd5e40bd909d7f75d"
server_key_version: "encryptionkey@0"
I20260808 04:02:48.346217 30701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance:
uuid: "64ab99f9f9b448d793e63c180fbe2af1"
format_stamp: "Formatted at 2026-08-08 04:02:48 on dist-test-slave-pww0"
server_key: "6c7fc329fb7cebfff79b01a5dac354fc"
server_key_iv: "a9d868339564815dd5e40bd909d7f75d"
server_key_version: "encryptionkey@0"
I20260808 04:02:48.351805 30701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260808 04:02:48.355698 30718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:48.357162 30701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:02:48.357323 30701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/wal
uuid: "64ab99f9f9b448d793e63c180fbe2af1"
format_stamp: "Formatted at 2026-08-08 04:02:48 on dist-test-slave-pww0"
server_key: "6c7fc329fb7cebfff79b01a5dac354fc"
server_key_iv: "a9d868339564815dd5e40bd909d7f75d"
server_key_version: "encryptionkey@0"
I20260808 04:02:48.357523 30701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:48.382637 30701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:48.383590 30701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:48.383894 30701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:48.384963 30701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:48.386591 30701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:48.386683 30701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:48.386787 30701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:48.386875 30701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:48.418913 30701 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.193:46299
I20260808 04:02:48.418968 30909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.193:46299 every 8 connection(s)
I20260808 04:02:48.420297 30701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/data/info.pb
I20260808 04:02:48.426237 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 30701
I20260808 04:02:48.426433 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-0/wal/instance
I20260808 04:02:48.426977 29679 external_mini_cluster.cc:1428] Setting key 4655e903d156c1d5ddb12b8ff0e97ed6
I20260808 04:02:48.430503 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.194:0
--local_ip_for_outbound_sockets=127.28.251.194
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41453
--builtin_ntp_servers=127.28.251.212:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:02:48.435175 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:48.423040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:44939 (local address 127.28.251.254:41453)
0808 04:02:48.423574 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:48.423579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:48.424999 (+ 1420us) server_negotiation.cc:407] Connection header received
0808 04:02:48.427390 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:48.427395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:48.427526 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:48.427636 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:48.429075 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:48.430037 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:48.432045 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:48.432674 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:48.432810 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:48.434331 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:48.434337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:48.434341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:48.434368 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:48.434419 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:48.434436 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:48.434524 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:48.434726 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 04:02:48.434836 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":283,"threads_started":1}
I20260808 04:02:48.436296 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:48.423715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41453 (local address 127.28.251.193:44939)
0808 04:02:48.424824 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:48.424863 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:48.427024 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:48.427899 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:48.427909 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:48.428017 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:48.428881 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:48.428914 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:48.430335 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:48.430340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:48.431863 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:48.431880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:48.432947 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:48.432953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:48.433177 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:48.433944 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:02:48.433947 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:48.433995 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:02:48.434096 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:48.434532 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:48.434538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:48.434545 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:02:48.434696 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 04:02:48.435065 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":279,"threads_started":1}
I20260808 04:02:48.437937 30912 heartbeater.cc:347] Connected to a master server at 127.28.251.254:41453
I20260808 04:02:48.438314 30912 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:48.439307 30912 heartbeater.cc:511] Master 127.28.251.254:41453 requested a full tablet report, sending...
I20260808 04:02:48.441787 30606 ts_manager.cc:194] Registered new tserver with Master: 64ab99f9f9b448d793e63c180fbe2af1 (127.28.251.193:46299)
I20260808 04:02:48.443356 30606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.193:44939
W20260808 04:02:48.795132 30919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:48.795483 30919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:48.795603 30919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:48.803097 30919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:48.803229 30919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:48.803342 30919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.194
I20260808 04:02:48.813078 30919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.251.194
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.194
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:48.814481 30919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:48.816501 30919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:48.833899 30924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:48.834740 30930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:48.834918 30919 server_base.cc:1034] running on GCE node
W20260808 04:02:48.835987 30926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:48.836715 30919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:48.837774 30919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:48.839025 30919 hybrid_clock.cc:648] HybridClock initialized: now 1786161768838962 us; error 46 us; skew 500 ppm
I20260808 04:02:48.839411 30919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:48.842020 30919 webserver.cc:460] Webserver started at http://127.28.251.194:33527/ using document root <none> and password file <none>
I20260808 04:02:48.842794 30919 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:48.842901 30919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:48.843242 30919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:48.846944 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/data/instance:
uuid: "bcf06e021c4a497caff9a051510bac8f"
format_stamp: "Formatted at 2026-08-08 04:02:48 on dist-test-slave-pww0"
server_key: "1e583cfa205b41f077d653a8e7664e06"
server_key_iv: "1982b334fbaa503a43d21a81919aae12"
server_key_version: "encryptionkey@0"
I20260808 04:02:48.847836 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance:
uuid: "bcf06e021c4a497caff9a051510bac8f"
format_stamp: "Formatted at 2026-08-08 04:02:48 on dist-test-slave-pww0"
server_key: "1e583cfa205b41f077d653a8e7664e06"
server_key_iv: "1982b334fbaa503a43d21a81919aae12"
server_key_version: "encryptionkey@0"
I20260808 04:02:48.853461 30919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:02:48.857462 30940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:48.858924 30919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:02:48.859123 30919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/wal
uuid: "bcf06e021c4a497caff9a051510bac8f"
format_stamp: "Formatted at 2026-08-08 04:02:48 on dist-test-slave-pww0"
server_key: "1e583cfa205b41f077d653a8e7664e06"
server_key_iv: "1982b334fbaa503a43d21a81919aae12"
server_key_version: "encryptionkey@0"
I20260808 04:02:48.859355 30919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:48.876294 30919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:48.877228 30919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:48.877550 30919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:48.878610 30919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:48.880263 30919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:48.880365 30919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:48.880477 30919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:48.880549 30919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:48.912560 30919 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.194:39785
I20260808 04:02:48.912633 31136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.194:39785 every 8 connection(s)
I20260808 04:02:48.913938 30919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/data/info.pb
I20260808 04:02:48.919021 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 30919
I20260808 04:02:48.919200 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/wal/instance
I20260808 04:02:48.919673 29679 external_mini_cluster.cc:1428] Setting key 347216d00a716bda5dfc7982cd4c642c
I20260808 04:02:48.923086 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.251.195:0
--local_ip_for_outbound_sockets=127.28.251.195
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41453
--builtin_ntp_servers=127.28.251.212:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:02:48.927697 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:48.916605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.194:50219 (local address 127.28.251.254:41453)
0808 04:02:48.916840 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:48.916846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:48.918307 (+ 1461us) server_negotiation.cc:407] Connection header received
0808 04:02:48.920278 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:48.920294 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:48.920391 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:48.920502 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:48.921941 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:48.923007 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:48.924736 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:48.925332 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:48.925462 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:48.926917 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:48.926924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:48.926928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:48.926951 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:48.926988 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:48.927003 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:48.927086 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:48.927288 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 04:02:48.927388 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 04:02:48.928748 31143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:48.917141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41453 (local address 127.28.251.194:50219)
0808 04:02:48.918094 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:48.918138 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:48.919984 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:48.920780 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:48.920792 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:48.920913 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:48.921758 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:48.921786 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:48.923183 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:48.923187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:48.924577 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:48.924591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:48.925585 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:48.925591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:48.925818 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:48.926574 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:02:48.926577 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:48.926621 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:02:48.926707 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:48.927094 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:48.927102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:48.927108 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:02:48.927257 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:02:48.927635 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":290,"threads_started":1}
I20260808 04:02:48.930246 31137 heartbeater.cc:347] Connected to a master server at 127.28.251.254:41453
I20260808 04:02:48.930619 31137 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:48.931681 31137 heartbeater.cc:511] Master 127.28.251.254:41453 requested a full tablet report, sending...
I20260808 04:02:48.933382 30606 ts_manager.cc:194] Registered new tserver with Master: bcf06e021c4a497caff9a051510bac8f (127.28.251.194:39785)
I20260808 04:02:48.934206 30606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.194:50219
W20260808 04:02:49.291345 31145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:49.291683 31145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:49.291774 31145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:49.299177 31145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:49.299304 31145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:49.299424 31145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.251.195
I20260808 04:02:49.308993 31145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.251.195
--webserver_port=0
--tserver_master_addrs=127.28.251.254:41453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.251.195
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:49.310411 31145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:49.312314 31145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:49.325616 31150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:49.325574 31157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:49.325541 31155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:49.326521 31145 server_base.cc:1034] running on GCE node
I20260808 04:02:49.327201 31145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:49.328351 31145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:49.329607 31145 hybrid_clock.cc:648] HybridClock initialized: now 1786161769329538 us; error 61 us; skew 500 ppm
I20260808 04:02:49.329985 31145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 04:02:49.332508 31145 webserver.cc:460] Webserver started at http://127.28.251.195:40079/ using document root <none> and password file <none>
I20260808 04:02:49.333235 31145 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:49.333344 31145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:49.333659 31145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:49.336761 31145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/data/instance:
uuid: "1a525cba294c435d803c11c3da63699a"
format_stamp: "Formatted at 2026-08-08 04:02:49 on dist-test-slave-pww0"
server_key: "9c7e2d166e8a480d66dc8ef39bc49be9"
server_key_iv: "de2b44e1839035343624f2d7cfeb8bc6"
server_key_version: "encryptionkey@0"
I20260808 04:02:49.337555 31145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance:
uuid: "1a525cba294c435d803c11c3da63699a"
format_stamp: "Formatted at 2026-08-08 04:02:49 on dist-test-slave-pww0"
server_key: "9c7e2d166e8a480d66dc8ef39bc49be9"
server_key_iv: "de2b44e1839035343624f2d7cfeb8bc6"
server_key_version: "encryptionkey@0"
I20260808 04:02:49.343144 31145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 04:02:49.346879 31174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:49.348563 31145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:02:49.348764 31145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/wal
uuid: "1a525cba294c435d803c11c3da63699a"
format_stamp: "Formatted at 2026-08-08 04:02:49 on dist-test-slave-pww0"
server_key: "9c7e2d166e8a480d66dc8ef39bc49be9"
server_key_iv: "de2b44e1839035343624f2d7cfeb8bc6"
server_key_version: "encryptionkey@0"
I20260808 04:02:49.349004 31145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:49.367697 31145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:49.368643 31145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:49.368955 31145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:49.369966 31145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:02:49.371519 31145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:02:49.371637 31145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:49.371759 31145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:02:49.371842 31145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:49.403234 31145 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.195:45441
I20260808 04:02:49.403291 31369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.195:45441 every 8 connection(s)
I20260808 04:02:49.404628 31145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/data/info.pb
I20260808 04:02:49.411762 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 31145
I20260808 04:02:49.411950 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/wal/instance
I20260808 04:02:49.412472 29679 external_mini_cluster.cc:1428] Setting key b654073c44a062274cf6a4d9b1eeb1c3
I20260808 04:02:49.418056 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.407346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.195:39783 (local address 127.28.251.254:41453)
0808 04:02:49.407603 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:49.407610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:49.409060 (+ 1450us) server_negotiation.cc:407] Connection header received
0808 04:02:49.410886 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:49.410891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:49.410964 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:49.411058 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:49.412271 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.413292 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.415076 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.415631 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.415749 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.417142 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:49.417147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:49.417151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:49.417170 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:49.417203 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:49.417214 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:49.417287 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:49.417524 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 04:02:49.417669 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:02:49.419202 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.407931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:41453 (local address 127.28.251.195:39783)
0808 04:02:49.408899 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:49.408937 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:49.410599 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:49.411261 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:49.411269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:49.411382 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:02:49.412076 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.412098 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.413461 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.413465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.414927 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.414941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.415870 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.415878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.416113 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.416799 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:02:49.416802 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:02:49.416850 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:02:49.416943 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:02:49.417305 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:02:49.417317 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:02:49.417324 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:02:49.417498 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 04:02:49.418010 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":302,"threads_started":1}
I20260808 04:02:49.420866 31370 heartbeater.cc:347] Connected to a master server at 127.28.251.254:41453
I20260808 04:02:49.421173 31370 heartbeater.cc:464] Registering TS with master...
I20260808 04:02:49.421959 31370 heartbeater.cc:511] Master 127.28.251.254:41453 requested a full tablet report, sending...
I20260808 04:02:49.423625 30606 ts_manager.cc:194] Registered new tserver with Master: 1a525cba294c435d803c11c3da63699a (127.28.251.195:45441)
I20260808 04:02:49.424530 30606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.251.195:39783
I20260808 04:02:49.428161 29679 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:02:49.440402 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.432239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.28.251.254:41453)
0808 04:02:49.432452 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:49.432460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:49.432850 (+ 390us) server_negotiation.cc:407] Connection header received
0808 04:02:49.432996 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:49.432999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:49.433064 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:49.433139 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:49.433568 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.434274 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.435581 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.436063 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.436161 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.439727 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:49.439732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:49.439735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:49.439757 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:49.439794 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:49.439807 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:49.439890 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:49.439981 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 04:02:49.440080 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:02:49.447105 30912 heartbeater.cc:503] Master 127.28.251.254:41453 was elected leader, sending a full tablet report...
I20260808 04:02:49.444859 30606 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:02:49.452250 30606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:02:49.477912 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.467449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.28.251.193:46299)
0808 04:02:49.470592 (+ 3143us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:49.470612 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:49.470645 (+ 33us) server_negotiation.cc:407] Connection header received
0808 04:02:49.470742 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:49.470750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:49.470959 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:49.471122 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:49.471727 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.473218 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.475659 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.476682 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.476901 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.477163 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:49.477380 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 04:02:49.477560 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":275,"threads_started":1}
I20260808 04:02:49.477905 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.466796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46299 (local address 127.0.0.1:59948)
0808 04:02:49.469338 (+ 2542us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:49.469358 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:49.469500 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:49.471130 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:49.471135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:49.471158 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:49.471533 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.471547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.473372 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.473377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.475299 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.475313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.476895 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.476906 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.477114 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.477168 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 04:02:49.477314 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 04:02:49.477522 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2407,"thread_start_us":1477,"threads_started":1}
I20260808 04:02:49.478775 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.466487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:39785 (local address 127.0.0.1:38022)
0808 04:02:49.468795 (+ 2308us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:49.468841 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:49.469073 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:49.470226 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:49.470234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:49.470261 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:49.470641 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.470655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.472554 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.472559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.475310 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.475323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.477834 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.477841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.478027 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.478066 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 04:02:49.478196 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 04:02:49.478331 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1972,"mutex_wait_us":137,"thread_start_us":2896,"threads_started":1}
I20260808 04:02:49.478989 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.466684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.28.251.194:39785)
0808 04:02:49.467335 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:49.467344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:49.469640 (+ 2296us) server_negotiation.cc:407] Connection header received
0808 04:02:49.469770 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:49.469778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:49.470017 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:49.470210 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:49.470851 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.472371 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.475748 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.477263 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.477445 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.477671 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:49.478341 (+ 670us) server_negotiation.cc:299] Negotiation successful
0808 04:02:49.478621 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":315,"threads_started":1}
I20260808 04:02:49.481487 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.468329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:45441 (local address 127.0.0.1:59088)
0808 04:02:49.469656 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:49.469673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:49.469797 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:49.472447 (+ 2650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:49.472452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:49.472479 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:49.473023 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.473036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.475135 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.475141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.477302 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.477317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.480704 (+ 3387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.480711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.480912 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.480947 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:02:49.481052 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 04:02:49.481199 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"thread_start_us":1127,"threads_started":1}
I20260808 04:02:49.481616 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.468955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59088 (local address 127.28.251.195:45441)
0808 04:02:49.471870 (+ 2915us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:49.471879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:49.471912 (+ 33us) server_negotiation.cc:407] Connection header received
0808 04:02:49.472010 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:49.472017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:49.472248 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:49.472458 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:49.473347 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.474973 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.479098 (+ 4125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.480469 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.480689 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.480923 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:49.481108 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 04:02:49.481292 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2724,"thread_start_us":2563,"threads_started":1}
I20260808 04:02:49.482807 30811 tablet_service.cc:1467] Processing CreateTablet for tablet f4a204b6df784882ac9875a5bc3cd759 (DEFAULT_TABLE table=test-table [id=1632628edd41443db3cd6fb439e2c0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:49.482803 31029 tablet_service.cc:1467] Processing CreateTablet for tablet f4a204b6df784882ac9875a5bc3cd759 (DEFAULT_TABLE table=test-table [id=1632628edd41443db3cd6fb439e2c0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:49.484313 30811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a204b6df784882ac9875a5bc3cd759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:49.484336 31029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a204b6df784882ac9875a5bc3cd759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:49.492722 31251 tablet_service.cc:1467] Processing CreateTablet for tablet f4a204b6df784882ac9875a5bc3cd759 (DEFAULT_TABLE table=test-table [id=1632628edd41443db3cd6fb439e2c0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:02:49.494567 31251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a204b6df784882ac9875a5bc3cd759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:49.496491 31400 tablet_bootstrap.cc:492] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: Bootstrap starting.
I20260808 04:02:49.499230 31402 tablet_bootstrap.cc:492] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1: Bootstrap starting.
I20260808 04:02:49.501467 31400 tablet_bootstrap.cc:654] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:49.502866 31400 log.cc:824] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:49.503141 31402 tablet_bootstrap.cc:654] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:49.504480 31402 log.cc:824] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:49.506601 31400 tablet_bootstrap.cc:492] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: No bootstrap required, opened a new log
I20260808 04:02:49.507028 31400 ts_tablet_manager.cc:1397] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260808 04:02:49.508237 31402 tablet_bootstrap.cc:492] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1: No bootstrap required, opened a new log
I20260808 04:02:49.508589 31402 ts_tablet_manager.cc:1397] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 04:02:49.509065 31409 tablet_bootstrap.cc:492] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: Bootstrap starting.
I20260808 04:02:49.512403 31409 tablet_bootstrap.cc:654] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:49.513509 31409 log.cc:824] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:49.516630 31409 tablet_bootstrap.cc:492] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: No bootstrap required, opened a new log
I20260808 04:02:49.516932 31409 ts_tablet_manager.cc:1397] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 04:02:49.518483 31400 raft_consensus.cc:348] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf06e021c4a497caff9a051510bac8f" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 39785 } } peers { permanent_uuid: "64ab99f9f9b448d793e63c180fbe2af1" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46299 } } peers { permanent_uuid: "1a525cba294c435d803c11c3da63699a" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45441 } }
I20260808 04:02:49.518993 31400 raft_consensus.cc:374] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:49.519166 31400 raft_consensus.cc:729] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf06e021c4a497caff9a051510bac8f, State: Initialized, Role: FOLLOWER
I20260808 04:02:49.519337 31402 raft_consensus.cc:348] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf06e021c4a497caff9a051510bac8f" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 39785 } } peers { permanent_uuid: "64ab99f9f9b448d793e63c180fbe2af1" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46299 } } peers { permanent_uuid: "1a525cba294c435d803c11c3da63699a" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45441 } }
I20260808 04:02:49.519906 31402 raft_consensus.cc:374] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:49.520051 31402 raft_consensus.cc:729] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ab99f9f9b448d793e63c180fbe2af1, State: Initialized, Role: FOLLOWER
I20260808 04:02:49.520048 31400 consensus_queue.cc:260] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf06e021c4a497caff9a051510bac8f" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 39785 } } peers { permanent_uuid: "64ab99f9f9b448d793e63c180fbe2af1" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46299 } } peers { permanent_uuid: "1a525cba294c435d803c11c3da63699a" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45441 } }
I20260808 04:02:49.520862 31402 consensus_queue.cc:260] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf06e021c4a497caff9a051510bac8f" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 39785 } } peers { permanent_uuid: "64ab99f9f9b448d793e63c180fbe2af1" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46299 } } peers { permanent_uuid: "1a525cba294c435d803c11c3da63699a" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45441 } }
I20260808 04:02:49.522878 31402 ts_tablet_manager.cc:1428] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260808 04:02:49.524111 31137 heartbeater.cc:503] Master 127.28.251.254:41453 was elected leader, sending a full tablet report...
I20260808 04:02:49.524820 31400 ts_tablet_manager.cc:1428] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260808 04:02:49.527077 31409 raft_consensus.cc:348] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf06e021c4a497caff9a051510bac8f" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 39785 } } peers { permanent_uuid: "64ab99f9f9b448d793e63c180fbe2af1" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46299 } } peers { permanent_uuid: "1a525cba294c435d803c11c3da63699a" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45441 } }
I20260808 04:02:49.527663 31409 raft_consensus.cc:374] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:49.527767 31409 raft_consensus.cc:729] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a525cba294c435d803c11c3da63699a, State: Initialized, Role: FOLLOWER
I20260808 04:02:49.528481 31409 consensus_queue.cc:260] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf06e021c4a497caff9a051510bac8f" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 39785 } } peers { permanent_uuid: "64ab99f9f9b448d793e63c180fbe2af1" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46299 } } peers { permanent_uuid: "1a525cba294c435d803c11c3da63699a" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45441 } }
I20260808 04:02:49.530169 31370 heartbeater.cc:503] Master 127.28.251.254:41453 was elected leader, sending a full tablet report...
I20260808 04:02:49.530938 31409 ts_tablet_manager.cc:1428] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260808 04:02:49.651391 31412 raft_consensus.cc:482] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:02:49.651818 31412 raft_consensus.cc:504] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf06e021c4a497caff9a051510bac8f" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 39785 } } peers { permanent_uuid: "64ab99f9f9b448d793e63c180fbe2af1" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46299 } } peers { permanent_uuid: "1a525cba294c435d803c11c3da63699a" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45441 } }
I20260808 04:02:49.653784 31412 leader_election.cc:288] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf06e021c4a497caff9a051510bac8f (127.28.251.194:39785), 1a525cba294c435d803c11c3da63699a (127.28.251.195:45441)
W20260808 04:02:49.656299 31371 tablet.cc:2369] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:02:49.661778 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.654437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:37377 (local address 127.28.251.194:39785)
0808 04:02:49.654857 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:49.654863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:49.655494 (+ 631us) server_negotiation.cc:407] Connection header received
0808 04:02:49.655671 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:49.655677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:49.655779 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:49.655886 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:49.656628 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.657582 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.659676 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.660604 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.660747 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.660798 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:49.661329 (+ 531us) server_negotiation.cc:299] Negotiation successful
0808 04:02:49.661474 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260808 04:02:49.661782 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.654407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.251.193:48805 (local address 127.28.251.195:45441)
0808 04:02:49.654642 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:49.654649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:49.655280 (+ 631us) server_negotiation.cc:407] Connection header received
0808 04:02:49.655504 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:49.655510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:49.655591 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:49.655680 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:02:49.656338 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.657179 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.659408 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.660393 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.660571 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.660630 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:49.661330 (+ 700us) server_negotiation.cc:299] Negotiation successful
0808 04:02:49.661474 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 04:02:49.661938 31415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.654436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.194:39785 (local address 127.28.251.193:37377)
0808 04:02:49.655336 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:49.655356 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:49.655543 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:49.655915 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:49.655918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:49.655943 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:49.656401 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.656415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.657716 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.657719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.659434 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.659449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.660808 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.660813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.661075 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.661170 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 04:02:49.661261 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 04:02:49.661483 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":125,"thread_start_us":556,"threads_started":1}
I20260808 04:02:49.661962 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.654187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.195:45441 (local address 127.28.251.193:48805)
0808 04:02:49.655128 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:49.655152 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 04:02:49.655328 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:02:49.655713 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:02:49.655718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:02:49.655743 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:02:49.656173 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.656185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.657319 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.657324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.659158 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:02:49.659176 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.660776 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.660786 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:02:49.661074 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.661168 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 04:02:49.661264 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 04:02:49.661483 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":201,"thread_start_us":507,"threads_started":1}
I20260808 04:02:49.663296 31296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a204b6df784882ac9875a5bc3cd759" candidate_uuid: "64ab99f9f9b448d793e63c180fbe2af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a525cba294c435d803c11c3da63699a" is_pre_election: true
I20260808 04:02:49.663348 31062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a204b6df784882ac9875a5bc3cd759" candidate_uuid: "64ab99f9f9b448d793e63c180fbe2af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf06e021c4a497caff9a051510bac8f" is_pre_election: true
I20260808 04:02:49.663892 31296 raft_consensus.cc:2454] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ab99f9f9b448d793e63c180fbe2af1 in term 0.
I20260808 04:02:49.663913 31062 raft_consensus.cc:2454] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ab99f9f9b448d793e63c180fbe2af1 in term 0.
I20260808 04:02:49.664603 30719 leader_election.cc:302] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a525cba294c435d803c11c3da63699a, 64ab99f9f9b448d793e63c180fbe2af1; no voters:
I20260808 04:02:49.665024 31412 raft_consensus.cc:2781] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:02:49.665167 31412 raft_consensus.cc:482] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:02:49.665267 31412 raft_consensus.cc:3037] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 0 FOLLOWER]: Advancing to term 1
W20260808 04:02:49.665972 31139 tablet.cc:2369] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:02:49.668105 31412 raft_consensus.cc:504] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf06e021c4a497caff9a051510bac8f" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 39785 } } peers { permanent_uuid: "64ab99f9f9b448d793e63c180fbe2af1" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46299 } } peers { permanent_uuid: "1a525cba294c435d803c11c3da63699a" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45441 } }
I20260808 04:02:49.668807 31412 leader_election.cc:288] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [CANDIDATE]: Term 1 election: Requested vote from peers bcf06e021c4a497caff9a051510bac8f (127.28.251.194:39785), 1a525cba294c435d803c11c3da63699a (127.28.251.195:45441)
I20260808 04:02:49.669348 31062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a204b6df784882ac9875a5bc3cd759" candidate_uuid: "64ab99f9f9b448d793e63c180fbe2af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf06e021c4a497caff9a051510bac8f"
I20260808 04:02:49.669405 31296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a204b6df784882ac9875a5bc3cd759" candidate_uuid: "64ab99f9f9b448d793e63c180fbe2af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a525cba294c435d803c11c3da63699a"
I20260808 04:02:49.669595 31062 raft_consensus.cc:3037] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:49.669607 31296 raft_consensus.cc:3037] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a [term 0 FOLLOWER]: Advancing to term 1
W20260808 04:02:49.672873 30913 tablet.cc:2369] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:02:49.673004 31062 raft_consensus.cc:2454] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ab99f9f9b448d793e63c180fbe2af1 in term 1.
I20260808 04:02:49.673004 31296 raft_consensus.cc:2454] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ab99f9f9b448d793e63c180fbe2af1 in term 1.
I20260808 04:02:49.673555 30724 leader_election.cc:302] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ab99f9f9b448d793e63c180fbe2af1, bcf06e021c4a497caff9a051510bac8f; no voters:
I20260808 04:02:49.673866 31412 raft_consensus.cc:2781] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:49.674319 31412 raft_consensus.cc:686] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 1 LEADER]: Becoming Leader. State: Replica: 64ab99f9f9b448d793e63c180fbe2af1, State: Running, Role: LEADER
I20260808 04:02:49.674999 31412 consensus_queue.cc:237] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf06e021c4a497caff9a051510bac8f" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 39785 } } peers { permanent_uuid: "64ab99f9f9b448d793e63c180fbe2af1" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46299 } } peers { permanent_uuid: "1a525cba294c435d803c11c3da63699a" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45441 } }
I20260808 04:02:49.680256 30606 catalog_manager.cc:5495] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64ab99f9f9b448d793e63c180fbe2af1 (127.28.251.193). New cstate: current_term: 1 leader_uuid: "64ab99f9f9b448d793e63c180fbe2af1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcf06e021c4a497caff9a051510bac8f" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 39785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64ab99f9f9b448d793e63c180fbe2af1" member_type: VOTER last_known_addr { host: "127.28.251.193" port: 46299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a525cba294c435d803c11c3da63699a" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45441 } health_report { overall_health: UNKNOWN } } }
I20260808 04:02:49.718662 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.713112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.28.251.193:46299)
0808 04:02:49.713347 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:49.713353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:49.713374 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:02:49.713532 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:49.713536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:49.713650 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:49.713791 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:02:49.714195 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.714927 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.716408 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.716849 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.716964 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.717510 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:02:49.718055 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:02:49.718202 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:49.718312 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 04:02:49.718403 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 04:02:49.727289 31296 raft_consensus.cc:1261] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a [term 1 FOLLOWER]: Refusing update from remote peer 64ab99f9f9b448d793e63c180fbe2af1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:49.727303 31062 raft_consensus.cc:1261] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f [term 1 FOLLOWER]: Refusing update from remote peer 64ab99f9f9b448d793e63c180fbe2af1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:02:49.728343 31416 consensus_queue.cc:1035] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a525cba294c435d803c11c3da63699a" member_type: VOTER last_known_addr { host: "127.28.251.195" port: 45441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:02:49.728722 31412 consensus_queue.cc:1035] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf06e021c4a497caff9a051510bac8f" member_type: VOTER last_known_addr { host: "127.28.251.194" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:02:49.742272 31421 mvcc.cc:205] Tried to move back new op lower bound from 7316118608792825856 to 7316118608592773120. Current Snapshot: MvccSnapshot[applied={T|T < 7316118608792825856}]
I20260808 04:02:49.756912 30603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:49.757208 30603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:02:49.761264 30603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bda789cf018a410bbfa7410abbfe47c4: Sending DeleteTablet for 3 replicas of tablet f4a204b6df784882ac9875a5bc3cd759
I20260808 04:02:49.762598 31029 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a204b6df784882ac9875a5bc3cd759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:49 UTC) from {username='slave'} at 127.0.0.1:38022
I20260808 04:02:49.762630 30811 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a204b6df784882ac9875a5bc3cd759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:49 UTC) from {username='slave'} at 127.0.0.1:59948
I20260808 04:02:49.763530 31251 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a204b6df784882ac9875a5bc3cd759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:02:49 UTC) from {username='slave'} at 127.0.0.1:59088
I20260808 04:02:49.764845 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 30701
I20260808 04:02:49.785105 31435 tablet_replica.cc:330] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: stopping tablet replica
I20260808 04:02:49.785594 31440 tablet_replica.cc:330] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: stopping tablet replica
I20260808 04:02:49.785590 31435 raft_consensus.cc:2229] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:49.786104 31435 raft_consensus.cc:2258] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:49.786077 31440 raft_consensus.cc:2229] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:02:49.786580 31440 raft_consensus.cc:2258] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:49.787541 31437 tablet_replica.cc:330] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1: stopping tablet replica
I20260808 04:02:49.789261 31435 ts_tablet_manager.cc:1905] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:49.789669 31440 ts_tablet_manager.cc:1905] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:02:49.794430 31435 ts_tablet_manager.cc:1918] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:49.794615 31435 log.cc:1196] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-1/wal/wals/f4a204b6df784882ac9875a5bc3cd759
I20260808 04:02:49.788841 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.766522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.28.251.193:46299)
0808 04:02:49.766756 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:49.766763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:49.777429 (+ 10666us) server_negotiation.cc:407] Connection header received
0808 04:02:49.778004 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:49.778012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:49.778154 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:49.778243 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:49.779055 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.780476 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.782632 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.783430 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.783560 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.786938 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:49.786956 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:49.786974 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:49.787880 (+ 906us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:49.787945 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:49.787973 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:49.788082 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:49.788324 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 04:02:49.788460 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:02:49.795426 31437 raft_consensus.cc:2229] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:02:49.795567 30902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:49.796792 31435 ts_tablet_manager.cc:1939] T f4a204b6df784882ac9875a5bc3cd759 P bcf06e021c4a497caff9a051510bac8f: Deleting consensus metadata
I20260808 04:02:49.797008 31440 ts_tablet_manager.cc:1918] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:02:49.797278 31440 log.cc:1196] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: Deleting WAL directory at /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786161430375636-29679-0/minicluster-data/ts-2/wal/wals/f4a204b6df784882ac9875a5bc3cd759
I20260808 04:02:49.797749 31440 ts_tablet_manager.cc:1939] T f4a204b6df784882ac9875a5bc3cd759 P 1a525cba294c435d803c11c3da63699a: Deleting consensus metadata
I20260808 04:02:49.799234 30580 catalog_manager.cc:4841] TS bcf06e021c4a497caff9a051510bac8f (127.28.251.194:39785): tablet f4a204b6df784882ac9875a5bc3cd759 (table test-table [id=1632628edd41443db3cd6fb439e2c0f9]) successfully deleted
I20260808 04:02:49.799356 30573 catalog_manager.cc:4841] TS 1a525cba294c435d803c11c3da63699a (127.28.251.195:45441): tablet f4a204b6df784882ac9875a5bc3cd759 (table test-table [id=1632628edd41443db3cd6fb439e2c0f9]) successfully deleted
I20260808 04:02:49.898380 31437 raft_consensus.cc:2258] T f4a204b6df784882ac9875a5bc3cd759 P 64ab99f9f9b448d793e63c180fbe2af1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:02:49.898792 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 30701
I20260808 04:02:49.929299 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 30919
W20260808 04:02:49.929306 30574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.251.193:46299 (error 108)
I20260808 04:02:49.931224 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.930769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46299 (local address 127.0.0.1:59974)
0808 04:02:49.930956 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:49.931071 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 04:02:49.931450 30574 catalog_manager.cc:4584] TS 64ab99f9f9b448d793e63c180fbe2af1 (127.28.251.193:46299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4a204b6df784882ac9875a5bc3cd759: Network error: Client connection negotiation failed: client connection to 127.28.251.193:46299: connect: Connection refused (error 111)
I20260808 04:02:49.936620 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.930150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.28.251.194:39785)
0808 04:02:49.930362 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:49.930369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:49.930497 (+ 128us) server_negotiation.cc:407] Connection header received
0808 04:02:49.930682 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:49.930687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:49.930769 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:49.930858 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:49.931280 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.932183 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.933544 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:49.933957 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:49.934076 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:49.935153 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:49.935162 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:49.935177 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:49.935848 (+ 671us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:49.935899 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:49.935921 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:49.936023 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:49.936163 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 04:02:49.936283 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:02:49.937366 31129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:49.985301 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.984726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46299 (local address 127.0.0.1:59984)
0808 04:02:49.984984 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:49.985105 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 04:02:49.986429 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:49.986027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46299 (local address 127.0.0.1:59996)
0808 04:02:49.986172 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:49.986263 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 04:02:50.028810 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 30919
I20260808 04:02:50.047160 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:50.046460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46299 (local address 127.0.0.1:60008)
0808 04:02:50.046789 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:50.046919 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260808 04:02:50.048346 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:50.047941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46299 (local address 127.0.0.1:60012)
0808 04:02:50.048102 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:50.048196 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 04:02:50.057355 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31145
I20260808 04:02:50.064139 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:50.058196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.28.251.195:45441)
0808 04:02:50.058395 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:50.058400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:50.058422 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:02:50.058531 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:50.058535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:50.058626 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:50.058723 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:50.059066 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:50.059855 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:50.061107 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:50.061563 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:50.061656 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:50.062713 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:50.062722 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:50.062737 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:50.063328 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:50.063378 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:50.063398 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:50.063659 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:50.063705 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 04:02:50.063812 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:02:50.064848 31365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:50.128974 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:50.128462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46299 (local address 127.0.0.1:60028)
0808 04:02:50.128689 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:50.128796 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260808 04:02:50.130139 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:50.129676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.193:46299 (local address 127.0.0.1:60040)
0808 04:02:50.129927 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 04:02:50.130010 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.193:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260808 04:02:50.154981 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 31145
I20260808 04:02:50.182796 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 30546
I20260808 04:02:50.184094 30652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:50.296758 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 30546
2026-08-08T04:02:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3061 ms)
[----------] 6 tests from EncryptionPolicyTest (24461 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:02:50 dist-test-slave-pww0 krb5kdc[31453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:02:50 dist-test-slave-pww0 krb5kdc[31453](info): set up 2 sockets
Aug 08 04:02:50 dist-test-slave-pww0 krb5kdc[31453](info): commencing operation
krb5kdc: starting...
W20260808 04:02:51.592581 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.003s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:02:51 dist-test-slave-pww0 krb5kdc[31453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:02:51.971788 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:52.344342 31481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:52.344676 31481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:52.344805 31481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:52.352608 31481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:02:52.352753 31481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:52.352824 31481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:52.352923 31481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:02:52.352977 31481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:02:52.363344 31481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46025
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:52.364758 31481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:52.366173 31481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:52.378525 31487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:52.378561 31493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:52.378489 31488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:52.379874 31481 server_base.cc:1034] running on GCE node
I20260808 04:02:52.380836 31481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:52.382452 31481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:52.383657 31481 hybrid_clock.cc:648] HybridClock initialized: now 1786161772383607 us; error 50 us; skew 500 ppm
I20260808 04:02:52.384050 31481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:52 dist-test-slave-pww0 krb5kdc[31453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:52.387214 31481 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:02:52.388893 31481 webserver.cc:460] Webserver started at http://127.0.0.1:46431/ using document root <none> and password file <none>
I20260808 04:02:52.389618 31481 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:52.389743 31481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:52.390049 31481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:52.393244 31481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "7a0a244f89d149d8ae521fdd6abd01ec"
format_stamp: "Formatted at 2026-08-08 04:02:52 on dist-test-slave-pww0"
server_key: "f306d97a02ac2048d84eb47fcf83794c"
server_key_iv: "f396b1f196d3122b373cb7df1bcf510d"
server_key_version: "encryptionkey@0"
I20260808 04:02:52.394059 31481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "7a0a244f89d149d8ae521fdd6abd01ec"
format_stamp: "Formatted at 2026-08-08 04:02:52 on dist-test-slave-pww0"
server_key: "f306d97a02ac2048d84eb47fcf83794c"
server_key_iv: "f396b1f196d3122b373cb7df1bcf510d"
server_key_version: "encryptionkey@0"
I20260808 04:02:52.399586 31481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 04:02:52.403512 31504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:52.405088 31481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:02:52.405299 31481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "7a0a244f89d149d8ae521fdd6abd01ec"
format_stamp: "Formatted at 2026-08-08 04:02:52 on dist-test-slave-pww0"
server_key: "f306d97a02ac2048d84eb47fcf83794c"
server_key_iv: "f396b1f196d3122b373cb7df1bcf510d"
server_key_version: "encryptionkey@0"
I20260808 04:02:52.405519 31481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:52.416813 31481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:52.421262 31481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:52.421561 31481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:52.440626 31481 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46025
I20260808 04:02:52.440673 31585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46025 every 8 connection(s)
I20260808 04:02:52.442049 31481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:02:52.446135 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:52.450951 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 31481
I20260808 04:02:52.451187 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:02:52.452082 29679 external_mini_cluster.cc:1428] Setting key d92cf35028860a62f2649e55e5a95366
I20260808 04:02:52.459168 31586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec: Bootstrap starting.
I20260808 04:02:52.463399 31586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:52.464738 31586 log.cc:824] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:52.468673 31586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec: No bootstrap required, opened a new log
Aug 08 04:02:52 dist-test-slave-pww0 krb5kdc[31453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:02:52.476886 31586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0a244f89d149d8ae521fdd6abd01ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } }
I20260808 04:02:52.477383 31586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:52.477500 31586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0a244f89d149d8ae521fdd6abd01ec, State: Initialized, Role: FOLLOWER
I20260808 04:02:52.478116 31586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0a244f89d149d8ae521fdd6abd01ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } }
I20260808 04:02:52.478389 31586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:02:52.478399 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:52.454524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43163 (local address 127.0.0.1:46025)
0808 04:02:52.455531 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:52.455546 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:52.455604 (+ 58us) server_negotiation.cc:407] Connection header received
0808 04:02:52.457564 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:52.457595 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:52.458081 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:52.458501 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:52.460561 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:52.462032 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:52.463941 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:52.464690 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:52.464844 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:52.469940 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:52.469949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:52.469966 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:52.470008 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:52.473394 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:52.474398 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:52.474406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:52.474414 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:52.474625 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:52.475139 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:52.475143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:52.475146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:52.475860 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:52.476215 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:52.476758 (+ 543us) server_negotiation.cc:299] Negotiation successful
0808 04:02:52.477113 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":427,"threads_started":1}
I20260808 04:02:52.478542 31586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:02:52.478745 31586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:52.481809 31586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0a244f89d149d8ae521fdd6abd01ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } }
I20260808 04:02:52.482405 31586 leader_election.cc:302] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a0a244f89d149d8ae521fdd6abd01ec; no voters:
I20260808 04:02:52.483032 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:02:52.483089 31586 leader_election.cc:288] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:02:52.483178 31601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:52.485302 31586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:02:52.492944 31601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [term 1 LEADER]: Becoming Leader. State: Replica: 7a0a244f89d149d8ae521fdd6abd01ec, State: Running, Role: LEADER
I20260808 04:02:52.493778 31601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0a244f89d149d8ae521fdd6abd01ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } }
I20260808 04:02:52.497103 31603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a0a244f89d149d8ae521fdd6abd01ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0a244f89d149d8ae521fdd6abd01ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } } }
I20260808 04:02:52.497578 31603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:52.498122 31606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a0a244f89d149d8ae521fdd6abd01ec. Latest consensus state: current_term: 1 leader_uuid: "7a0a244f89d149d8ae521fdd6abd01ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0a244f89d149d8ae521fdd6abd01ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46025 } } }
I20260808 04:02:52.498404 31606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:52.500717 31635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:02:52.508996 31635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:02:52.514519 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:52.494969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51961 (local address 127.0.0.1:46025)
0808 04:02:52.495160 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:52.495167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:52.495854 (+ 687us) server_negotiation.cc:407] Connection header received
0808 04:02:52.496195 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:52.496201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:52.496290 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:52.496401 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:52.498720 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:52.500005 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:52.501390 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:52.502124 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:52.502272 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:52.506027 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:52.506034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:52.506039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:52.506083 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:52.510135 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:52.511564 (+ 1429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:52.511570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:52.511573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:52.511755 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:52.512415 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:52.512420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:52.512424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:52.512878 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:52.513148 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:52.513896 (+ 748us) server_negotiation.cc:299] Negotiation successful
0808 04:02:52.514117 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:02:52.522153 31635 catalog_manager.cc:1347] Generated new cluster ID: a5c2e402813a42a9b770ae3708ca16d7
I20260808 04:02:52.522292 31635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:02:52.535339 31635 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:02:52.536746 31635 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:02:52.553529 31635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a0a244f89d149d8ae521fdd6abd01ec: Generated new TSK 0
I20260808 04:02:52.554484 31635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:02:52.562832 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31481
I20260808 04:02:52.564571 31580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:52.658378 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 31481
W20260808 04:02:52.684434 31450 connection.cc:535] client connection to 127.0.0.1:46025 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46025): Connection reset by peer (error 104)
2026-08-08T04:02:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:02:52 dist-test-slave-pww0 krb5kdc[31649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:02:52 dist-test-slave-pww0 krb5kdc[31649](info): set up 2 sockets
Aug 08 04:02:52 dist-test-slave-pww0 krb5kdc[31649](info): commencing operation
krb5kdc: starting...
W20260808 04:02:53.956678 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.006s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:02:54 dist-test-slave-pww0 krb5kdc[31649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:02:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:02:54.329177 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:54.693492 31668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:54.693837 31668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:54.693991 31668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:54.701525 31668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:02:54.701687 31668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:54.701773 31668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:54.701869 31668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:02:54.701928 31668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:02:54.712877 31668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:54.714251 31668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:54.715749 31668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:54.728526 31678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:54.728590 31680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:54.728567 31683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:54.729372 31668 server_base.cc:1034] running on GCE node
I20260808 04:02:54.730260 31668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:54.732056 31668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:54.733314 31668 hybrid_clock.cc:648] HybridClock initialized: now 1786161774733255 us; error 58 us; skew 500 ppm
I20260808 04:02:54.733747 31668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:54 dist-test-slave-pww0 krb5kdc[31649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:54.740239 31668 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:02:54.742069 31668 webserver.cc:460] Webserver started at http://127.0.0.1:36453/ using document root <none> and password file <none>
I20260808 04:02:54.742823 31668 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:54.742960 31668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:54.743286 31668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:54.746699 31668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "7c96d72593b64df7838899652578ee20"
format_stamp: "Formatted at 2026-08-08 04:02:54 on dist-test-slave-pww0"
server_key: "93a119a994d5b99875e6edf9352f5447"
server_key_iv: "4f10732b77e2378e3846b826a5e3c477"
server_key_version: "encryptionkey@0"
I20260808 04:02:54.747725 31668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "7c96d72593b64df7838899652578ee20"
format_stamp: "Formatted at 2026-08-08 04:02:54 on dist-test-slave-pww0"
server_key: "93a119a994d5b99875e6edf9352f5447"
server_key_iv: "4f10732b77e2378e3846b826a5e3c477"
server_key_version: "encryptionkey@0"
I20260808 04:02:54.753808 31668 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 04:02:54.757956 31696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:54.759544 31668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:02:54.759738 31668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "7c96d72593b64df7838899652578ee20"
format_stamp: "Formatted at 2026-08-08 04:02:54 on dist-test-slave-pww0"
server_key: "93a119a994d5b99875e6edf9352f5447"
server_key_iv: "4f10732b77e2378e3846b826a5e3c477"
server_key_version: "encryptionkey@0"
I20260808 04:02:54.759953 31668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:54.780190 31668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:54.784582 31668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:54.784879 31668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:54.803963 31668 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34259
I20260808 04:02:54.803992 31785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34259 every 8 connection(s)
I20260808 04:02:54.805531 31668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:02:54.807101 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 31668
I20260808 04:02:54.807297 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:02:54.807811 29679 external_mini_cluster.cc:1428] Setting key b98b3383beff93b25fccc7d31f057e6d
I20260808 04:02:54.810283 31786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:54.822293 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:54.809658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58913 (local address 127.0.0.1:34259)
0808 04:02:54.810512 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:54.810527 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:54.810571 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:02:54.812322 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:54.812348 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:54.812768 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:54.813181 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:54.813892 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:54.815301 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:54.817263 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:54.818026 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:54.818169 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:54.819639 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:54.819648 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:54.819665 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:54.820350 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:54.820449 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:54.820481 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:54.820619 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:54.820938 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 04:02:54.821159 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":361,"threads_started":1}
I20260808 04:02:54.823828 31786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20: Bootstrap starting.
I20260808 04:02:54.826668 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:02:54.827366 31786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:54.828471 31786 log.cc:824] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:54.832227 31786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20: No bootstrap required, opened a new log
I20260808 04:02:54.837610 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:54.831176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52497 (local address 127.0.0.1:34259)
0808 04:02:54.831334 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:54.831339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:54.831891 (+ 552us) server_negotiation.cc:407] Connection header received
0808 04:02:54.832096 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:54.832100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:54.832164 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:54.832257 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:54.832802 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:54.833585 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:54.834929 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:54.835432 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:54.835606 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:54.836841 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:54.836846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:54.836849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:54.836871 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:54.836916 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:54.836932 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:54.837042 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:54.837238 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 04:02:54.837306 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:02:54.840703 31786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c96d72593b64df7838899652578ee20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260808 04:02:54.841154 31786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:54.841256 31786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c96d72593b64df7838899652578ee20, State: Initialized, Role: FOLLOWER
I20260808 04:02:54.842001 31786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c96d72593b64df7838899652578ee20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260808 04:02:54.842265 31786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:02:54.842413 31786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:02:54.842617 31786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:54.845063 31786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c96d72593b64df7838899652578ee20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260808 04:02:54.845600 31786 leader_election.cc:302] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c96d72593b64df7838899652578ee20; no voters:
I20260808 04:02:54.846215 31786 leader_election.cc:288] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:02:54.846361 31804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:54.847040 31804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [term 1 LEADER]: Becoming Leader. State: Replica: 7c96d72593b64df7838899652578ee20, State: Running, Role: LEADER
I20260808 04:02:54.847684 31804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c96d72593b64df7838899652578ee20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260808 04:02:54.848522 31786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:02:54.850792 31807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c96d72593b64df7838899652578ee20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c96d72593b64df7838899652578ee20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } } }
I20260808 04:02:54.850996 31808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c96d72593b64df7838899652578ee20. Latest consensus state: current_term: 1 leader_uuid: "7c96d72593b64df7838899652578ee20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c96d72593b64df7838899652578ee20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } } }
I20260808 04:02:54.851257 31807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:54.851257 31808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:54.852164 31811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:02:54.859112 31811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:02:54.870631 31811 catalog_manager.cc:1347] Generated new cluster ID: 041ad5da09804acda6add8fbae00175d
I20260808 04:02:54.870771 31811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:02:54.903772 31811 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:02:54.905820 31811 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:02:54.921888 31811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c96d72593b64df7838899652578ee20: Generated new TSK 0
I20260808 04:02:54.922930 31811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:02:54.943835 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31668
W20260808 04:02:54.945384 31777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58913
W20260808 04:02:54.946435 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31668: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:02:54.946607 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 31668
W20260808 04:02:54.971244 31645 connection.cc:535] client connection to 127.0.0.1:34259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34259): Connection reset by peer (error 104)
2026-08-08T04:02:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:02:55 dist-test-slave-pww0 krb5kdc[31834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:02:55 dist-test-slave-pww0 krb5kdc[31834](info): set up 2 sockets
Aug 08 04:02:55 dist-test-slave-pww0 krb5kdc[31834](info): commencing operation
krb5kdc: starting...
W20260808 04:02:56.233127 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.004s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:02:56 dist-test-slave-pww0 krb5kdc[31834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:02:56.604426 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:57.000789 31856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:57.001116 31856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:57.001256 31856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:57.008828 31856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:02:57.008963 31856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:57.009045 31856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:57.009127 31856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:02:57.009194 31856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:02:57.019709 31856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33553
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:57.021143 31856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:57.022475 31856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:57.034716 31866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:57.034802 31870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:57.034785 31863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:57.035400 31856 server_base.cc:1034] running on GCE node
I20260808 04:02:57.036324 31856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:57.037945 31856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:57.039198 31856 hybrid_clock.cc:648] HybridClock initialized: now 1786161777039130 us; error 53 us; skew 500 ppm
I20260808 04:02:57.039680 31856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:57 dist-test-slave-pww0 krb5kdc[31834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:57.043166 31856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:02:57.044904 31856 webserver.cc:460] Webserver started at http://127.0.0.1:45037/ using document root <none> and password file <none>
I20260808 04:02:57.045648 31856 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:57.045792 31856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:57.046116 31856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:57.049422 31856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "70c3ef1620a141c89462363773c5fe11"
format_stamp: "Formatted at 2026-08-08 04:02:57 on dist-test-slave-pww0"
server_key: "539fe709eeb97253607f0973204db9e0"
server_key_iv: "9d7e020db6dd03df3e8ff80fb398915a"
server_key_version: "encryptionkey@0"
I20260808 04:02:57.050283 31856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "70c3ef1620a141c89462363773c5fe11"
format_stamp: "Formatted at 2026-08-08 04:02:57 on dist-test-slave-pww0"
server_key: "539fe709eeb97253607f0973204db9e0"
server_key_iv: "9d7e020db6dd03df3e8ff80fb398915a"
server_key_version: "encryptionkey@0"
I20260808 04:02:57.056641 31856 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260808 04:02:57.060799 31879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:57.062317 31856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:02:57.062525 31856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "70c3ef1620a141c89462363773c5fe11"
format_stamp: "Formatted at 2026-08-08 04:02:57 on dist-test-slave-pww0"
server_key: "539fe709eeb97253607f0973204db9e0"
server_key_iv: "9d7e020db6dd03df3e8ff80fb398915a"
server_key_version: "encryptionkey@0"
I20260808 04:02:57.062743 31856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:57.069160 31856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:57.073552 31856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:57.073858 31856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:57.095850 31856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33553
I20260808 04:02:57.095897 31961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33553 every 8 connection(s)
I20260808 04:02:57.097368 31856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:02:57.101478 31962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:57.105844 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 31856
I20260808 04:02:57.106040 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:02:57.106494 29679 external_mini_cluster.cc:1428] Setting key 79b5cd23c49358794a5523590a6793ca
I20260808 04:02:57.114998 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:57.108507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41591 (local address 127.0.0.1:33553)
0808 04:02:57.109407 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:57.109421 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:57.109469 (+ 48us) server_negotiation.cc:407] Connection header received
0808 04:02:57.111221 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:57.111251 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:57.111737 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:57.112166 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:57.113295 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:57.113302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:57.113317 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:57.113333 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:57.113402 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:57.113423 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:57.113645 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:57.113792 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 04:02:57.113995 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":389,"threads_started":1}
I20260808 04:02:57.115154 31962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11: Bootstrap starting.
I20260808 04:02:57.118944 31962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:57.119215 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:02:57.120165 31962 log.cc:824] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:57.123363 31962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11: No bootstrap required, opened a new log
I20260808 04:02:57.126456 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:57.123713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57441 (local address 127.0.0.1:33553)
0808 04:02:57.123884 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:57.123889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:57.124390 (+ 501us) server_negotiation.cc:407] Connection header received
0808 04:02:57.124598 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:57.124604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:57.124663 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:57.124751 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:57.125763 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:57.125773 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:57.125777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:02:57.125795 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:57.125853 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:02:57.125869 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:57.125987 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:57.126102 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 04:02:57.126173 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:02:57.134613 31962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3ef1620a141c89462363773c5fe11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33553 } }
I20260808 04:02:57.135033 31962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:57.135113 31962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c3ef1620a141c89462363773c5fe11, State: Initialized, Role: FOLLOWER
I20260808 04:02:57.135764 31962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3ef1620a141c89462363773c5fe11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33553 } }
I20260808 04:02:57.136003 31962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:02:57.136113 31962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:02:57.136246 31962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:57.138597 31962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3ef1620a141c89462363773c5fe11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33553 } }
I20260808 04:02:57.139140 31962 leader_election.cc:302] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70c3ef1620a141c89462363773c5fe11; no voters:
I20260808 04:02:57.139793 31962 leader_election.cc:288] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:02:57.139952 31982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:57.140586 31982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [term 1 LEADER]: Becoming Leader. State: Replica: 70c3ef1620a141c89462363773c5fe11, State: Running, Role: LEADER
I20260808 04:02:57.141124 31982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3ef1620a141c89462363773c5fe11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33553 } }
I20260808 04:02:57.141959 31962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:02:57.144155 31985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70c3ef1620a141c89462363773c5fe11. Latest consensus state: current_term: 1 leader_uuid: "70c3ef1620a141c89462363773c5fe11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3ef1620a141c89462363773c5fe11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33553 } } }
I20260808 04:02:57.144565 31985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:57.143771 31983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70c3ef1620a141c89462363773c5fe11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c3ef1620a141c89462363773c5fe11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33553 } } }
I20260808 04:02:57.145033 31983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11 [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:57.147993 31992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:02:57.153002 31992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:02:57.164077 31992 catalog_manager.cc:1347] Generated new cluster ID: c89ae9bd0aac4e15a25f06d7ccc6571d
I20260808 04:02:57.164208 31992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:02:57.180943 31992 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:02:57.181434 31992 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:02:57.188337 31992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70c3ef1620a141c89462363773c5fe11: Generated new TSK 0
I20260808 04:02:57.189529 31992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:02:57.202876 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31856
W20260808 04:02:57.204326 31954 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41591
W20260808 04:02:57.205133 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 31856: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:02:57.205286 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 31856
2026-08-08T04:02:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:02:57 dist-test-slave-pww0 krb5kdc[32008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:02:57 dist-test-slave-pww0 krb5kdc[32008](info): set up 2 sockets
Aug 08 04:02:57 dist-test-slave-pww0 krb5kdc[32008](info): commencing operation
krb5kdc: starting...
W20260808 04:02:58.495968 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.011s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:02:58 dist-test-slave-pww0 krb5kdc[32008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:02:58.866633 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:02:59.240309 32026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:02:59.240638 32026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:02:59.240772 32026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:02:59.248227 32026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:02:59.248358 32026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:02:59.248413 32026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:02:59.248500 32026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:02:59.248553 32026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:02:59.259274 32026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:02:59.260740 32026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:02:59.262100 32026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:02:59.274495 32034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:59.274533 32036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:02:59.274463 32038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:02:59.275504 32026 server_base.cc:1034] running on GCE node
I20260808 04:02:59.276353 32026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:02:59.277834 32026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:02:59.279042 32026 hybrid_clock.cc:648] HybridClock initialized: now 1786161779278990 us; error 40 us; skew 500 ppm
I20260808 04:02:59.279407 32026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:02:59 dist-test-slave-pww0 krb5kdc[32008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:02:59.282542 32026 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:02:59.284127 32026 webserver.cc:460] Webserver started at http://127.0.0.1:42835/ using document root <none> and password file <none>
I20260808 04:02:59.284818 32026 fs_manager.cc:359] Metadata directory not provided
I20260808 04:02:59.284951 32026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:02:59.285245 32026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:02:59.288421 32026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "6ac33b1e7cb8450cb47375a5492b676b"
format_stamp: "Formatted at 2026-08-08 04:02:59 on dist-test-slave-pww0"
server_key: "3f3ba3a4166e3c2bf63b7c122b4d76ec"
server_key_iv: "7ae11a279c1f2cf63fd054ff9b726535"
server_key_version: "encryptionkey@0"
I20260808 04:02:59.289230 32026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "6ac33b1e7cb8450cb47375a5492b676b"
format_stamp: "Formatted at 2026-08-08 04:02:59 on dist-test-slave-pww0"
server_key: "3f3ba3a4166e3c2bf63b7c122b4d76ec"
server_key_iv: "7ae11a279c1f2cf63fd054ff9b726535"
server_key_version: "encryptionkey@0"
I20260808 04:02:59.294939 32026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:02:59.299191 32049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:02:59.300717 32026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260808 04:02:59.300912 32026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "6ac33b1e7cb8450cb47375a5492b676b"
format_stamp: "Formatted at 2026-08-08 04:02:59 on dist-test-slave-pww0"
server_key: "3f3ba3a4166e3c2bf63b7c122b4d76ec"
server_key_iv: "7ae11a279c1f2cf63fd054ff9b726535"
server_key_version: "encryptionkey@0"
I20260808 04:02:59.301119 32026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:02:59.325237 32026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:02:59.329967 32026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:02:59.330269 32026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:02:59.347946 32026 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46753
I20260808 04:02:59.347981 32130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46753 every 8 connection(s)
I20260808 04:02:59.349468 32026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:02:59.353111 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 32026
I20260808 04:02:59.353287 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:02:59.353672 32132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:02:59.353729 29679 external_mini_cluster.cc:1428] Setting key 1511898e3c441601dc11563801675cc6
I20260808 04:02:59.369087 32132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b: Bootstrap starting.
Aug 08 04:02:59 dist-test-slave-pww0 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:02:59.373063 32132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b: Neither blocks nor log segments found. Creating new log.
I20260808 04:02:59.374281 32132 log.cc:824] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:02:59.377318 32132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b: No bootstrap required, opened a new log
I20260808 04:02:59.378916 32138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:59.355572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53597 (local address 127.0.0.1:46753)
0808 04:02:59.356408 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:59.356420 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:59.356465 (+ 45us) server_negotiation.cc:407] Connection header received
0808 04:02:59.358568 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:59.358593 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:59.359056 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:59.359434 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:59.361609 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:59.362958 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:59.364841 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:59.365652 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:59.365812 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:59.371177 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:59.371187 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:59.371204 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:59.371252 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:59.374563 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:59.375425 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:59.375434 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:59.375441 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:59.375671 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:59.376084 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:59.376089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:59.376091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:59.376651 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:59.376957 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:59.377422 (+ 465us) server_negotiation.cc:299] Negotiation successful
0808 04:02:59.377894 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":363,"threads_started":1}
I20260808 04:02:59.383131 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:02:59.386356 32132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac33b1e7cb8450cb47375a5492b676b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260808 04:02:59.386772 32132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:02:59.386879 32132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac33b1e7cb8450cb47375a5492b676b, State: Initialized, Role: FOLLOWER
I20260808 04:02:59.387560 32132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac33b1e7cb8450cb47375a5492b676b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260808 04:02:59.387933 32132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:02:59.388110 32132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:02:59.388295 32132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:02:59.390676 32132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac33b1e7cb8450cb47375a5492b676b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260808 04:02:59.391225 32132 leader_election.cc:302] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ac33b1e7cb8450cb47375a5492b676b; no voters:
I20260808 04:02:59.391907 32132 leader_election.cc:288] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:02:59.392010 32149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:02:59.394273 32149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [term 1 LEADER]: Becoming Leader. State: Replica: 6ac33b1e7cb8450cb47375a5492b676b, State: Running, Role: LEADER
I20260808 04:02:59.394404 32132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:02:59.394866 32149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac33b1e7cb8450cb47375a5492b676b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260808 04:02:59.400449 32152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ac33b1e7cb8450cb47375a5492b676b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac33b1e7cb8450cb47375a5492b676b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } } }
I20260808 04:02:59.400871 32152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:59.401356 32154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ac33b1e7cb8450cb47375a5492b676b. Latest consensus state: current_term: 1 leader_uuid: "6ac33b1e7cb8450cb47375a5492b676b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ac33b1e7cb8450cb47375a5492b676b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } } }
I20260808 04:02:59.401654 32154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b [sys.catalog]: This master's current role is: LEADER
I20260808 04:02:59.405292 32138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:02:59.387775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38697 (local address 127.0.0.1:46753)
0808 04:02:59.388016 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 04:02:59.388022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:02:59.388517 (+ 495us) server_negotiation.cc:407] Connection header received
0808 04:02:59.388716 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:02:59.388721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:02:59.388795 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:02:59.388882 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:02:59.390369 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:59.391146 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:59.392706 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:02:59.393306 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:02:59.393390 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:02:59.396225 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:02:59.396235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:02:59.396240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:02:59.396286 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:02:59.400212 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:59.401332 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:59.401337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:59.401345 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:59.401533 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:02:59.402223 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:02:59.402229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:02:59.402232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:02:59.402728 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:02:59.404236 (+ 1508us) server_negotiation.cc:1035] Waiting for connection context
0808 04:02:59.404447 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 04:02:59.404751 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 04:02:59.405879 32168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:02:59.412158 32168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:02:59.421248 32168 catalog_manager.cc:1347] Generated new cluster ID: 2c543624706046c6b811257698e94d1a
I20260808 04:02:59.421378 32168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:02:59.437057 32168 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:02:59.438443 32168 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:02:59.447638 32168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ac33b1e7cb8450cb47375a5492b676b: Generated new TSK 0
I20260808 04:02:59.448570 32168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:02:59.455103 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 32026
I20260808 04:02:59.456504 32126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:02:59.548424 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 32026
2026-08-08T04:02:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:02:59 dist-test-slave-pww0 krb5kdc[32186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:02:59 dist-test-slave-pww0 krb5kdc[32186](info): set up 2 sockets
Aug 08 04:02:59 dist-test-slave-pww0 krb5kdc[32186](info): commencing operation
krb5kdc: starting...
W20260808 04:03:00.832008 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.002s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:01 dist-test-slave-pww0 krb5kdc[32186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:03:01.210556 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:01.574324 32203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:01.574672 32203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:01.574812 32203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:01.582366 32203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:01.582518 32203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:01.582604 32203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:01.582701 32203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:01.582760 32203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:01.593400 32203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:01.594821 32203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:01.596275 32203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:01.608736 32211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:01.608667 32215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:01.608798 32210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:01.609822 32203 server_base.cc:1034] running on GCE node
I20260808 04:03:01.610743 32203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:01.612404 32203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:01.613626 32203 hybrid_clock.cc:648] HybridClock initialized: now 1786161781613582 us; error 36 us; skew 500 ppm
I20260808 04:03:01.613996 32203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:01 dist-test-slave-pww0 krb5kdc[32186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:01.621266 32203 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:03:01.623000 32203 webserver.cc:460] Webserver started at http://127.0.0.1:40663/ using document root <none> and password file <none>
I20260808 04:03:01.623797 32203 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:01.623922 32203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:01.624202 32203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:01.627365 32203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "87bdb00958ca4975beec9fffbb87d7a5"
format_stamp: "Formatted at 2026-08-08 04:03:01 on dist-test-slave-pww0"
server_key: "6796c64c0b2acf14d48cc5953a82a4a5"
server_key_iv: "ef8f9d337cc2fa33edbbf383ad60b042"
server_key_version: "encryptionkey@0"
I20260808 04:03:01.628193 32203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "87bdb00958ca4975beec9fffbb87d7a5"
format_stamp: "Formatted at 2026-08-08 04:03:01 on dist-test-slave-pww0"
server_key: "6796c64c0b2acf14d48cc5953a82a4a5"
server_key_iv: "ef8f9d337cc2fa33edbbf383ad60b042"
server_key_version: "encryptionkey@0"
I20260808 04:03:01.633560 32203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:03:01.637369 32223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:01.638842 32203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 04:03:01.639019 32203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "87bdb00958ca4975beec9fffbb87d7a5"
format_stamp: "Formatted at 2026-08-08 04:03:01 on dist-test-slave-pww0"
server_key: "6796c64c0b2acf14d48cc5953a82a4a5"
server_key_iv: "ef8f9d337cc2fa33edbbf383ad60b042"
server_key_version: "encryptionkey@0"
I20260808 04:03:01.639231 32203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:01.666324 32203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:01.671054 32203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:01.671347 32203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:01.689697 32203 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37927
I20260808 04:03:01.689731 32309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37927 every 8 connection(s)
I20260808 04:03:01.691277 32203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:01.695391 32310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:01.698585 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 32203
I20260808 04:03:01.698787 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:01.699237 29679 external_mini_cluster.cc:1428] Setting key 4dbcec662100e53efea6efbf10a88e8f
I20260808 04:03:01.708138 32310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5: Bootstrap starting.
I20260808 04:03:01.712127 32310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:01.713436 32310 log.cc:824] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:01.714089 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:01.701260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47087 (local address 127.0.0.1:37927)
0808 04:03:01.702057 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:01.702069 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:01.702115 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:03:01.703902 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:01.703933 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:01.704400 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:01.704784 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:01.705529 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:01.707004 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:01.708993 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:01.709779 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:01.709949 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:01.711351 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:01.711360 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:01.711379 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:01.712064 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:01.712159 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:01.712188 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:01.712322 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:01.712589 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 04:03:01.712918 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":347,"threads_started":1}
I20260808 04:03:01.716902 32310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5: No bootstrap required, opened a new log
I20260808 04:03:01.718410 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:01.725674 32310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bdb00958ca4975beec9fffbb87d7a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260808 04:03:01.726194 32310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:01.726303 32310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87bdb00958ca4975beec9fffbb87d7a5, State: Initialized, Role: FOLLOWER
I20260808 04:03:01.726914 32310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bdb00958ca4975beec9fffbb87d7a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260808 04:03:01.727144 32310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:01.727280 32310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:01.727443 32310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:01.729920 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:01.722927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55747 (local address 127.0.0.1:37927)
0808 04:03:01.723140 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:01.723148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:01.723677 (+ 529us) server_negotiation.cc:407] Connection header received
0808 04:03:01.723863 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:01.723869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:01.723961 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:01.724090 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:01.724715 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:01.725752 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:01.727450 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:01.727988 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:01.728120 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:01.729202 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:01.729207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:01.729211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:01.729231 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:01.729274 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:01.729298 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:01.729376 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:01.729519 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 04:03:01.729615 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:03:01.729943 32310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bdb00958ca4975beec9fffbb87d7a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260808 04:03:01.730623 32310 leader_election.cc:302] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87bdb00958ca4975beec9fffbb87d7a5; no voters:
I20260808 04:03:01.731272 32310 leader_election.cc:288] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:01.731431 32324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:01.732201 32324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 87bdb00958ca4975beec9fffbb87d7a5, State: Running, Role: LEADER
I20260808 04:03:01.732759 32324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bdb00958ca4975beec9fffbb87d7a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } }
I20260808 04:03:01.733574 32310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:01.735524 32327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87bdb00958ca4975beec9fffbb87d7a5. Latest consensus state: current_term: 1 leader_uuid: "87bdb00958ca4975beec9fffbb87d7a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bdb00958ca4975beec9fffbb87d7a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } } }
I20260808 04:03:01.736037 32327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:01.735504 32326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87bdb00958ca4975beec9fffbb87d7a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87bdb00958ca4975beec9fffbb87d7a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37927 } } }
I20260808 04:03:01.736470 32326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:01.737536 32331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:01.742816 32331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:01.753831 32331 catalog_manager.cc:1347] Generated new cluster ID: f927de191c214458a443efe69f399187
I20260808 04:03:01.753970 32331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:01.770134 32331 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:01.772135 32331 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:01.784631 32331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87bdb00958ca4975beec9fffbb87d7a5: Generated new TSK 0
I20260808 04:03:01.785895 32331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:01.803557 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 32203
W20260808 04:03:01.804934 32300 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47087
W20260808 04:03:01.805790 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 32203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:01.805941 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 32203
2026-08-08T04:03:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2259 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:01 dist-test-slave-pww0 krb5kdc[32362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:01 dist-test-slave-pww0 krb5kdc[32362](info): set up 2 sockets
Aug 08 04:03:01 dist-test-slave-pww0 krb5kdc[32362](info): commencing operation
krb5kdc: starting...
W20260808 04:03:03.102897 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.009s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:03 dist-test-slave-pww0 krb5kdc[32362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:03:03.488210 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:03.858552 32380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:03.858876 32380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:03.858971 32380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:03.866495 32380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:03.866638 32380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:03.866688 32380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:03.866751 32380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:03.866796 32380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:03.877322 32380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:03.878750 32380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:03.880174 32380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:03.892823 32386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:03.892863 32388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:03.893052 32392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:03.893864 32380 server_base.cc:1034] running on GCE node
I20260808 04:03:03.894702 32380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:03.896271 32380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:03.897521 32380 hybrid_clock.cc:648] HybridClock initialized: now 1786161783897465 us; error 45 us; skew 500 ppm
I20260808 04:03:03.897910 32380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:03 dist-test-slave-pww0 krb5kdc[32362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:03.901252 32380 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:03:03.902935 32380 webserver.cc:460] Webserver started at http://127.0.0.1:34671/ using document root <none> and password file <none>
I20260808 04:03:03.903729 32380 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:03.903860 32380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:03.904176 32380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:03.907320 32380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "f00689e141b74846806f7669191af66e"
format_stamp: "Formatted at 2026-08-08 04:03:03 on dist-test-slave-pww0"
server_key: "4dc2cdf004c4fd9a99a804cf57007df3"
server_key_iv: "ddc3ade530e804168eea5d533c3c1853"
server_key_version: "encryptionkey@0"
I20260808 04:03:03.908174 32380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "f00689e141b74846806f7669191af66e"
format_stamp: "Formatted at 2026-08-08 04:03:03 on dist-test-slave-pww0"
server_key: "4dc2cdf004c4fd9a99a804cf57007df3"
server_key_iv: "ddc3ade530e804168eea5d533c3c1853"
server_key_version: "encryptionkey@0"
I20260808 04:03:03.913635 32380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:03:03.917526 32402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:03.918926 32380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:03:03.919106 32380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "f00689e141b74846806f7669191af66e"
format_stamp: "Formatted at 2026-08-08 04:03:03 on dist-test-slave-pww0"
server_key: "4dc2cdf004c4fd9a99a804cf57007df3"
server_key_iv: "ddc3ade530e804168eea5d533c3c1853"
server_key_version: "encryptionkey@0"
I20260808 04:03:03.919314 32380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:03.925853 32380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:03.930115 32380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:03.930392 32380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:03.948868 32380 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40321
I20260808 04:03:03.948870 32490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40321 every 8 connection(s)
I20260808 04:03:03.950407 32380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:03.954572 32492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:03.956905 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 32380
I20260808 04:03:03.957104 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:03.957671 29679 external_mini_cluster.cc:1428] Setting key 67e8e7da2eeed7b0b3822ee57d2a57d9
I20260808 04:03:03.966156 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:03.959696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56315 (local address 127.0.0.1:40321)
0808 04:03:03.960592 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:03.960605 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:03.960657 (+ 52us) server_negotiation.cc:407] Connection header received
0808 04:03:03.962419 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:03.962446 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:03.962840 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:03.963197 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:03.964399 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:03.964406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:03.964421 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:03.964437 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:03.964512 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:03.964536 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:03.964732 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:03.964839 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 04:03:03.965051 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":411,"threads_started":1}
I20260808 04:03:03.967978 32492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e: Bootstrap starting.
I20260808 04:03:03.970533 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:03.971700 32492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:03.972800 32492 log.cc:824] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:03.976475 32492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e: No bootstrap required, opened a new log
I20260808 04:03:03.977754 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:03.975055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49549 (local address 127.0.0.1:40321)
0808 04:03:03.975207 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:03.975212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:03.975754 (+ 542us) server_negotiation.cc:407] Connection header received
0808 04:03:03.975953 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:03.975958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:03.976011 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:03.976092 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:03.977105 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:03.977110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:03.977113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:03.977129 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:03.977172 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:03.977192 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:03.977283 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:03.977431 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 04:03:03.977485 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:03:03.985301 32492 raft_consensus.cc:348] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00689e141b74846806f7669191af66e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } }
I20260808 04:03:03.985734 32492 raft_consensus.cc:374] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:03.985844 32492 raft_consensus.cc:729] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00689e141b74846806f7669191af66e, State: Initialized, Role: FOLLOWER
I20260808 04:03:03.986440 32492 consensus_queue.cc:260] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00689e141b74846806f7669191af66e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } }
I20260808 04:03:03.986686 32492 raft_consensus.cc:388] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:03.986835 32492 raft_consensus.cc:482] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:03.986999 32492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:03.989434 32492 raft_consensus.cc:504] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00689e141b74846806f7669191af66e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } }
I20260808 04:03:03.990007 32492 leader_election.cc:302] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f00689e141b74846806f7669191af66e; no voters:
I20260808 04:03:03.990759 32492 leader_election.cc:288] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:03.990904 32516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:03.992044 32516 raft_consensus.cc:686] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [term 1 LEADER]: Becoming Leader. State: Replica: f00689e141b74846806f7669191af66e, State: Running, Role: LEADER
I20260808 04:03:03.992586 32516 consensus_queue.cc:237] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00689e141b74846806f7669191af66e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } }
I20260808 04:03:03.993021 32492 sys_catalog.cc:564] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:03.995630 32517 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f00689e141b74846806f7669191af66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00689e141b74846806f7669191af66e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } } }
I20260808 04:03:03.996088 32517 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:03.996225 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f00689e141b74846806f7669191af66e. Latest consensus state: current_term: 1 leader_uuid: "f00689e141b74846806f7669191af66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f00689e141b74846806f7669191af66e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } } }
I20260808 04:03:03.996526 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:03.997323 32526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:04.002980 32526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:04.013069 32526 catalog_manager.cc:1347] Generated new cluster ID: 61024d079a7849598c544c80623328cf
I20260808 04:03:04.013206 32526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:04.032649 32526 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:04.033191 32526 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:04.043597 32526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f00689e141b74846806f7669191af66e: Generated new TSK 0
I20260808 04:03:04.044883 32526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:04.060328 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 32380
W20260808 04:03:04.062013 32484 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56315
W20260808 04:03:04.062893 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 32380: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:04.063071 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 32380
2026-08-08T04:03:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2267 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:04 dist-test-slave-pww0 krb5kdc[32545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:04 dist-test-slave-pww0 krb5kdc[32545](info): set up 2 sockets
Aug 08 04:03:04 dist-test-slave-pww0 krb5kdc[32545](info): commencing operation
krb5kdc: starting...
W20260808 04:03:05.374962 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.001s sys 0.063s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:05 dist-test-slave-pww0 krb5kdc[32545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:03:05.751068 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:06.122999 32573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:06.123358 32573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:06.123517 32573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:06.131361 32573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:06.131560 32573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:06.131654 32573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:06.131745 32573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:06.131819 32573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:06.142547 32573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:06.144073 32573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:06.145527 32573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:06.158293 32585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:06.158208 32588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:06.158248 32586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:06.159256 32573 server_base.cc:1034] running on GCE node
I20260808 04:03:06.160207 32573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:06.161830 32573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:06.163059 32573 hybrid_clock.cc:648] HybridClock initialized: now 1786161786163003 us; error 46 us; skew 500 ppm
I20260808 04:03:06.163486 32573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:06 dist-test-slave-pww0 krb5kdc[32545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:06.166777 32573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:03:06.168457 32573 webserver.cc:460] Webserver started at http://127.0.0.1:45871/ using document root <none> and password file <none>
I20260808 04:03:06.169265 32573 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:06.169407 32573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:06.169759 32573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:06.173118 32573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "6f1ffb85ebe044b59c67c3332d8dc3a4"
format_stamp: "Formatted at 2026-08-08 04:03:06 on dist-test-slave-pww0"
server_key: "f01a02814a5e2a13b26ca222d1815d93"
server_key_iv: "841a37c890e643ae78bd4e36ec2a6d1a"
server_key_version: "encryptionkey@0"
I20260808 04:03:06.173981 32573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "6f1ffb85ebe044b59c67c3332d8dc3a4"
format_stamp: "Formatted at 2026-08-08 04:03:06 on dist-test-slave-pww0"
server_key: "f01a02814a5e2a13b26ca222d1815d93"
server_key_iv: "841a37c890e643ae78bd4e36ec2a6d1a"
server_key_version: "encryptionkey@0"
I20260808 04:03:06.179772 32573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 04:03:06.183811 32600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:06.185356 32573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:03:06.185554 32573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "6f1ffb85ebe044b59c67c3332d8dc3a4"
format_stamp: "Formatted at 2026-08-08 04:03:06 on dist-test-slave-pww0"
server_key: "f01a02814a5e2a13b26ca222d1815d93"
server_key_iv: "841a37c890e643ae78bd4e36ec2a6d1a"
server_key_version: "encryptionkey@0"
I20260808 04:03:06.185776 32573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:06.204999 32573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:06.209338 32573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:06.209620 32573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:06.228122 32573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44067
I20260808 04:03:06.228156 32696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44067 every 8 connection(s)
I20260808 04:03:06.229605 32573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:06.230067 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 32573
I20260808 04:03:06.230252 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:06.230875 29679 external_mini_cluster.cc:1428] Setting key da3028ab6074003998468808fbab77b9
I20260808 04:03:06.234364 32697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 04:03:06 dist-test-slave-pww0 krb5kdc[32545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:03:06.246358 32697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4: Bootstrap starting.
I20260808 04:03:06.250233 32697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:06.251748 32697 log.cc:824] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:06.255489 32697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4: No bootstrap required, opened a new log
I20260808 04:03:06.255533 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:06.232916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:47251 (local address 127.0.0.1:44067)
0808 04:03:06.233706 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:06.233749 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:06.233795 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:03:06.235568 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:06.235594 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:06.236014 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:06.236392 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:06.237950 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:06.239386 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:06.240766 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:06.241482 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:06.241662 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:06.246853 (+ 5191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:06.246863 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:06.246883 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:06.246930 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:06.250233 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:06.251451 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:06.251508 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:06.251516 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:06.251736 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:06.252309 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:06.252313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:06.252315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:06.253194 (+ 879us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:06.253502 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:06.254078 (+ 576us) server_negotiation.cc:299] Negotiation successful
0808 04:03:06.254403 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":322,"threads_started":1}
I20260808 04:03:06.259793 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:06.265362 32697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1ffb85ebe044b59c67c3332d8dc3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44067 } }
I20260808 04:03:06.265892 32697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:06.265995 32697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1ffb85ebe044b59c67c3332d8dc3a4, State: Initialized, Role: FOLLOWER
I20260808 04:03:06.266700 32697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1ffb85ebe044b59c67c3332d8dc3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44067 } }
I20260808 04:03:06.266973 32697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:06.267107 32697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:06.267280 32697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:06.270298 32697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1ffb85ebe044b59c67c3332d8dc3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44067 } }
I20260808 04:03:06.270968 32697 leader_election.cc:302] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f1ffb85ebe044b59c67c3332d8dc3a4; no voters:
I20260808 04:03:06.271767 32697 leader_election.cc:288] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:06.271927 32715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:06.272675 32715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [term 1 LEADER]: Becoming Leader. State: Replica: 6f1ffb85ebe044b59c67c3332d8dc3a4, State: Running, Role: LEADER
I20260808 04:03:06.273450 32715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1ffb85ebe044b59c67c3332d8dc3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44067 } }
I20260808 04:03:06.274369 32697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:06.277016 32718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f1ffb85ebe044b59c67c3332d8dc3a4. Latest consensus state: current_term: 1 leader_uuid: "6f1ffb85ebe044b59c67c3332d8dc3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1ffb85ebe044b59c67c3332d8dc3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44067 } } }
I20260808 04:03:06.277447 32718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:06.279080 32716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f1ffb85ebe044b59c67c3332d8dc3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1ffb85ebe044b59c67c3332d8dc3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44067 } } }
I20260808 04:03:06.279359 32716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:06.280377 32727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:06.283421 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:06.265447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:58947 (local address 127.0.0.1:44067)
0808 04:03:06.265644 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:06.265650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:06.266017 (+ 367us) server_negotiation.cc:407] Connection header received
0808 04:03:06.266219 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:06.266224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:06.266313 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:06.266443 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:06.268158 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:06.269349 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:06.270775 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:06.271430 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:06.271615 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:06.275018 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:06.275025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:06.275030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:06.275079 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:06.279497 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:06.280712 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:06.280720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:06.280724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:06.280910 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:06.281508 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:06.281514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:06.281517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:06.281966 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:06.282321 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:06.282705 (+ 384us) server_negotiation.cc:299] Negotiation successful
0808 04:03:06.282936 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 04:03:06.287153 32727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:06.298621 32727 catalog_manager.cc:1347] Generated new cluster ID: 9df86af2096e49fd9f75264f44957ee9
I20260808 04:03:06.298766 32727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:06.318370 32727 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:06.320514 32727 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:06.334034 32727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f1ffb85ebe044b59c67c3332d8dc3a4: Generated new TSK 0
I20260808 04:03:06.335270 32727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:06.344843 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 32573
I20260808 04:03:06.346484 32689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:06.440066 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 32573
W20260808 04:03:06.467129 32543 connection.cc:535] client connection to 127.0.0.1:44067 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44067): Connection reset by peer (error 104)
2026-08-08T04:03:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:06 dist-test-slave-pww0 krb5kdc[32758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:06 dist-test-slave-pww0 krb5kdc[32758](info): set up 2 sockets
Aug 08 04:03:06 dist-test-slave-pww0 krb5kdc[32758](info): commencing operation
krb5kdc: starting...
W20260808 04:03:07.744509 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.002s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:07 dist-test-slave-pww0 krb5kdc[32758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:03:08.126395 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:08.489944 311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:08.490279 311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:08.490397 311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:08.497924 311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:08.498061 311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:08.498116 311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:08.498178 311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:08.498221 311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:08.508669 311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:08.510066 311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:08.511489 311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:08.523646 325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:08.523607 330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:08.523684 327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:08.525393 311 server_base.cc:1034] running on GCE node
I20260808 04:03:08.526311 311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:08.528048 311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:08.529299 311 hybrid_clock.cc:648] HybridClock initialized: now 1786161788529233 us; error 57 us; skew 500 ppm
I20260808 04:03:08.529723 311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:08 dist-test-slave-pww0 krb5kdc[32758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:08.533411 311 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:03:08.535159 311 webserver.cc:460] Webserver started at http://127.0.0.1:37197/ using document root <none> and password file <none>
I20260808 04:03:08.535993 311 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:08.536135 311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:08.536470 311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:08.539834 311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "cd7849512d37402db373f558b3216511"
format_stamp: "Formatted at 2026-08-08 04:03:08 on dist-test-slave-pww0"
server_key: "8e297a1527a1b2f4646471566e1ec9b1"
server_key_iv: "e5a0f0d687068ff7197309a45d9b7917"
server_key_version: "encryptionkey@0"
I20260808 04:03:08.540678 311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "cd7849512d37402db373f558b3216511"
format_stamp: "Formatted at 2026-08-08 04:03:08 on dist-test-slave-pww0"
server_key: "8e297a1527a1b2f4646471566e1ec9b1"
server_key_iv: "e5a0f0d687068ff7197309a45d9b7917"
server_key_version: "encryptionkey@0"
I20260808 04:03:08.546295 311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:03:08.550197 339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:08.551702 311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 04:03:08.551893 311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "cd7849512d37402db373f558b3216511"
format_stamp: "Formatted at 2026-08-08 04:03:08 on dist-test-slave-pww0"
server_key: "8e297a1527a1b2f4646471566e1ec9b1"
server_key_iv: "e5a0f0d687068ff7197309a45d9b7917"
server_key_version: "encryptionkey@0"
I20260808 04:03:08.552102 311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:08.573349 311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:08.578054 311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:08.578349 311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:08.597076 311 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45351
I20260808 04:03:08.597114 431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45351 every 8 connection(s)
I20260808 04:03:08.598546 311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:08.602645 432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:08.604355 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 311
I20260808 04:03:08.604521 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:08.604986 29679 external_mini_cluster.cc:1428] Setting key a403503f0d8b98de4e4e5b7c4434e39b
I20260808 04:03:08.616770 432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511: Bootstrap starting.
I20260808 04:03:08.618934 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:08.606921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:46365 (local address 127.0.0.1:45351)
0808 04:03:08.607835 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:08.607847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:08.607900 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:03:08.609580 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:08.609609 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:08.610101 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:08.610547 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:08.611288 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:08.612846 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:08.614257 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:08.615007 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:08.615198 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:08.616548 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:08.616556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:08.616572 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:08.617172 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:08.617260 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:08.617287 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:08.617436 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:08.617736 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 04:03:08.617909 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":410,"threads_started":1}
I20260808 04:03:08.620239 432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:08.621426 432 log.cc:824] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:08.623034 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:08.624497 432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511: No bootstrap required, opened a new log
I20260808 04:03:08.632483 432 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7849512d37402db373f558b3216511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260808 04:03:08.632915 432 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:08.633016 432 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7849512d37402db373f558b3216511, State: Initialized, Role: FOLLOWER
I20260808 04:03:08.633618 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:08.627439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:56605 (local address 127.0.0.1:45351)
0808 04:03:08.627666 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:08.627672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:08.627978 (+ 306us) server_negotiation.cc:407] Connection header received
0808 04:03:08.628186 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:08.628190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:08.628254 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:08.628346 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:08.628868 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:08.629664 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:08.631009 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:08.631529 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:08.631654 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:08.632823 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:08.632830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:08.632835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:08.632854 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:08.632908 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:08.632928 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:08.633045 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:08.633243 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 04:03:08.633305 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:03:08.633896 432 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7849512d37402db373f558b3216511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260808 04:03:08.634153 432 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:08.634307 432 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:08.634479 432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:08.637060 432 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7849512d37402db373f558b3216511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260808 04:03:08.637645 432 leader_election.cc:302] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd7849512d37402db373f558b3216511; no voters:
I20260808 04:03:08.638429 432 leader_election.cc:288] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:08.638490 443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:08.639834 443 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [term 1 LEADER]: Becoming Leader. State: Replica: cd7849512d37402db373f558b3216511, State: Running, Role: LEADER
I20260808 04:03:08.640416 443 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7849512d37402db373f558b3216511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260808 04:03:08.640718 432 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:08.645042 444 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd7849512d37402db373f558b3216511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7849512d37402db373f558b3216511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } } }
I20260808 04:03:08.645066 446 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7849512d37402db373f558b3216511. Latest consensus state: current_term: 1 leader_uuid: "cd7849512d37402db373f558b3216511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7849512d37402db373f558b3216511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } } }
I20260808 04:03:08.645534 444 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:08.645589 446 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:08.647256 461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:08.652245 461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:08.661578 461 catalog_manager.cc:1347] Generated new cluster ID: ffd0d959cef7483db88e9e52c3db6c7d
I20260808 04:03:08.661702 461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:08.674467 461 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:08.676038 461 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:08.687558 461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd7849512d37402db373f558b3216511: Generated new TSK 0
I20260808 04:03:08.688804 461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:08.702497 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 311
W20260808 04:03:08.704298 425 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.195:46365
W20260808 04:03:08.705207 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 311: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:08.705353 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 311
W20260808 04:03:08.731014 32755 connection.cc:535] client connection to 127.0.0.1:45351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45351): Connection reset by peer (error 104)
2026-08-08T04:03:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2263 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:08 dist-test-slave-pww0 krb5kdc[482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:08 dist-test-slave-pww0 krb5kdc[482](info): set up 2 sockets
Aug 08 04:03:08 dist-test-slave-pww0 krb5kdc[482](info): commencing operation
krb5kdc: starting...
W20260808 04:03:09.993189 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.000s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:10 dist-test-slave-pww0 krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:03:10.371387 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:10.741030 503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:10.741354 503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:10.741475 503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:10.748816 503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:10.748950 503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:10.749025 503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:10.749127 503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:10.749198 503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:10.759541 503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:10.760998 503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:10.762377 503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:10.774986 512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:10.775048 513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:10.775079 516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:10.775902 503 server_base.cc:1034] running on GCE node
I20260808 04:03:10.776760 503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:10.778366 503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:10.779587 503 hybrid_clock.cc:648] HybridClock initialized: now 1786161790779521 us; error 56 us; skew 500 ppm
I20260808 04:03:10.780016 503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:10 dist-test-slave-pww0 krb5kdc[482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:10.783202 503 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:03:10.784890 503 webserver.cc:460] Webserver started at http://127.0.0.1:35491/ using document root <none> and password file <none>
I20260808 04:03:10.785627 503 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:10.785760 503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:10.786098 503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:10.789415 503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "8127a2ab3fab4360b4fcd12987a71325"
format_stamp: "Formatted at 2026-08-08 04:03:10 on dist-test-slave-pww0"
server_key: "54c92854eecb97609aaf925893104c88"
server_key_iv: "7a1df646139dde6355a50d9c2f76db09"
server_key_version: "encryptionkey@0"
I20260808 04:03:10.790225 503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "8127a2ab3fab4360b4fcd12987a71325"
format_stamp: "Formatted at 2026-08-08 04:03:10 on dist-test-slave-pww0"
server_key: "54c92854eecb97609aaf925893104c88"
server_key_iv: "7a1df646139dde6355a50d9c2f76db09"
server_key_version: "encryptionkey@0"
I20260808 04:03:10.795943 503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 04:03:10.799714 527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:10.801164 503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 04:03:10.801370 503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "8127a2ab3fab4360b4fcd12987a71325"
format_stamp: "Formatted at 2026-08-08 04:03:10 on dist-test-slave-pww0"
server_key: "54c92854eecb97609aaf925893104c88"
server_key_iv: "7a1df646139dde6355a50d9c2f76db09"
server_key_version: "encryptionkey@0"
I20260808 04:03:10.801577 503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:10.807528 503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:10.812073 503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:10.812378 503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:10.829422 503 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37669
I20260808 04:03:10.829444 617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37669 every 8 connection(s)
I20260808 04:03:10.830981 503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:10.835312 619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:10.839743 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 503
I20260808 04:03:10.839915 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:10.840379 29679 external_mini_cluster.cc:1428] Setting key 7ee3027ec4e1bd4ab085b872b93a66a2
I20260808 04:03:10.848148 619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325: Bootstrap starting.
I20260808 04:03:10.849924 623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:10.842403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:47701 (local address 127.0.0.1:37669)
0808 04:03:10.843369 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:10.843386 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:10.843445 (+ 59us) server_negotiation.cc:407] Connection header received
0808 04:03:10.846187 (+ 2742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:10.846227 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:10.846647 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:10.847015 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:10.848201 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:10.848207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:10.848238 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:10.848253 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:10.848320 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:10.848356 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:10.848550 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:10.848677 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 04:03:10.848883 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":410,"threads_started":1}
I20260808 04:03:10.852102 619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:10.853477 619 log.cc:824] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:10.854254 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:10.857079 619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325: No bootstrap required, opened a new log
I20260808 04:03:10.862437 623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:10.859599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:58561 (local address 127.0.0.1:37669)
0808 04:03:10.859821 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:10.859833 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:10.860401 (+ 568us) server_negotiation.cc:407] Connection header received
0808 04:03:10.860591 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:10.860597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:10.860648 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:10.860742 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:10.861734 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:10.861740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:10.861748 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:10.861765 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:10.861811 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:10.861832 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:10.861943 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:10.862073 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 04:03:10.862141 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:03:10.866007 619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8127a2ab3fab4360b4fcd12987a71325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37669 } }
I20260808 04:03:10.866415 619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:10.866524 619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8127a2ab3fab4360b4fcd12987a71325, State: Initialized, Role: FOLLOWER
I20260808 04:03:10.867115 619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8127a2ab3fab4360b4fcd12987a71325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37669 } }
I20260808 04:03:10.867355 619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:10.867511 619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:10.867693 619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:10.870150 619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8127a2ab3fab4360b4fcd12987a71325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37669 } }
I20260808 04:03:10.870715 619 leader_election.cc:302] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8127a2ab3fab4360b4fcd12987a71325; no voters:
I20260808 04:03:10.871322 619 leader_election.cc:288] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:10.871505 639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:10.872154 639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [term 1 LEADER]: Becoming Leader. State: Replica: 8127a2ab3fab4360b4fcd12987a71325, State: Running, Role: LEADER
I20260808 04:03:10.872694 639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8127a2ab3fab4360b4fcd12987a71325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37669 } }
I20260808 04:03:10.873471 619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:10.875639 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8127a2ab3fab4360b4fcd12987a71325. Latest consensus state: current_term: 1 leader_uuid: "8127a2ab3fab4360b4fcd12987a71325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8127a2ab3fab4360b4fcd12987a71325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37669 } } }
I20260808 04:03:10.876127 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:10.877135 649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:10.877014 640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8127a2ab3fab4360b4fcd12987a71325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8127a2ab3fab4360b4fcd12987a71325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37669 } } }
I20260808 04:03:10.878571 640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:10.883175 649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:10.893532 649 catalog_manager.cc:1347] Generated new cluster ID: ce7e220f470946139b40dbedd1686221
I20260808 04:03:10.893666 649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:10.921614 649 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:10.922207 649 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:10.942373 649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8127a2ab3fab4360b4fcd12987a71325: Generated new TSK 0
I20260808 04:03:10.943671 649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:10.954289 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 503
W20260808 04:03:10.956030 610 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.195:47701
W20260808 04:03:10.956773 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 503: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:10.956919 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 503
2026-08-08T04:03:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2251 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:11 dist-test-slave-pww0 krb5kdc[680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:11 dist-test-slave-pww0 krb5kdc[680](info): set up 2 sockets
Aug 08 04:03:11 dist-test-slave-pww0 krb5kdc[680](info): commencing operation
krb5kdc: starting...
W20260808 04:03:12.255079 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.000s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:12 dist-test-slave-pww0 krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:03:12.634614 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:13.000594 699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:13.000960 699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:13.001094 699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:13.008706 699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:13.008841 699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:13.008929 699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:13.009001 699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:13.009039 699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:13.019361 699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40229
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:13.020800 699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:13.022199 699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:13.034459 705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:13.034497 716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:13.034421 711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:13.035374 699 server_base.cc:1034] running on GCE node
I20260808 04:03:13.036263 699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:13.037952 699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:13.039199 699 hybrid_clock.cc:648] HybridClock initialized: now 1786161793039148 us; error 51 us; skew 500 ppm
I20260808 04:03:13.039656 699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:13 dist-test-slave-pww0 krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:13.050175 699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:03:13.052026 699 webserver.cc:460] Webserver started at http://127.0.0.1:43353/ using document root <none> and password file <none>
I20260808 04:03:13.052798 699 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:13.052943 699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:13.053278 699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:13.056739 699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "796f099f29aa48d1aa15aade37ee0168"
format_stamp: "Formatted at 2026-08-08 04:03:13 on dist-test-slave-pww0"
server_key: "1e8fd3fc845b7b73270aade0eb92d46d"
server_key_iv: "1e639ad26c4f8ac7c63378011d312221"
server_key_version: "encryptionkey@0"
I20260808 04:03:13.057587 699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "796f099f29aa48d1aa15aade37ee0168"
format_stamp: "Formatted at 2026-08-08 04:03:13 on dist-test-slave-pww0"
server_key: "1e8fd3fc845b7b73270aade0eb92d46d"
server_key_iv: "1e639ad26c4f8ac7c63378011d312221"
server_key_version: "encryptionkey@0"
I20260808 04:03:13.063632 699 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 04:03:13.067915 726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:13.069509 699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:03:13.069729 699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "796f099f29aa48d1aa15aade37ee0168"
format_stamp: "Formatted at 2026-08-08 04:03:13 on dist-test-slave-pww0"
server_key: "1e8fd3fc845b7b73270aade0eb92d46d"
server_key_iv: "1e639ad26c4f8ac7c63378011d312221"
server_key_version: "encryptionkey@0"
I20260808 04:03:13.069949 699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:13.085691 699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:13.090404 699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:13.090703 699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:13.109733 699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40229
I20260808 04:03:13.109752 823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40229 every 8 connection(s)
I20260808 04:03:13.111243 699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:13.114491 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 699
I20260808 04:03:13.114672 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:13.115249 29679 external_mini_cluster.cc:1428] Setting key 34a5f9d6ae7151590d2087cac1b8fe47
I20260808 04:03:13.115701 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:13.131893 824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168: Bootstrap starting.
Aug 08 04:03:13 dist-test-slave-pww0 krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:03:13.136112 824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:13.137307 824 log.cc:824] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:13.140620 824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168: No bootstrap required, opened a new log
I20260808 04:03:13.143644 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:13.117200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:37667 (local address 127.0.0.1:40229)
0808 04:03:13.117995 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:13.118007 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:13.118050 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:03:13.120567 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:13.120594 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:13.121090 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:13.121488 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:13.123610 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:13.125348 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:13.127502 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:13.128764 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:13.128997 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:13.134739 (+ 5742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:13.134750 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:13.134767 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:13.134817 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:13.138408 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:13.139556 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:13.139564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:13.139571 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:13.139789 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:13.140384 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:13.140389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:13.140391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:13.141037 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:13.141395 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:13.142013 (+ 618us) server_negotiation.cc:299] Negotiation successful
0808 04:03:13.142385 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":335,"threads_started":1}
I20260808 04:03:13.148357 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:13.149405 824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f099f29aa48d1aa15aade37ee0168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40229 } }
I20260808 04:03:13.149804 824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:13.149904 824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796f099f29aa48d1aa15aade37ee0168, State: Initialized, Role: FOLLOWER
I20260808 04:03:13.150504 824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f099f29aa48d1aa15aade37ee0168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40229 } }
I20260808 04:03:13.150741 824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:13.150877 824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:13.151060 824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:13.153697 824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f099f29aa48d1aa15aade37ee0168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40229 } }
I20260808 04:03:13.154352 824 leader_election.cc:302] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 796f099f29aa48d1aa15aade37ee0168; no voters:
I20260808 04:03:13.155092 824 leader_election.cc:288] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:13.155226 839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:13.157693 824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:13.162302 839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [term 1 LEADER]: Becoming Leader. State: Replica: 796f099f29aa48d1aa15aade37ee0168, State: Running, Role: LEADER
I20260808 04:03:13.163017 839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f099f29aa48d1aa15aade37ee0168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40229 } }
I20260808 04:03:13.167397 841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "796f099f29aa48d1aa15aade37ee0168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f099f29aa48d1aa15aade37ee0168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40229 } } }
I20260808 04:03:13.167579 843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796f099f29aa48d1aa15aade37ee0168. Latest consensus state: current_term: 1 leader_uuid: "796f099f29aa48d1aa15aade37ee0168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796f099f29aa48d1aa15aade37ee0168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40229 } } }
I20260808 04:03:13.167915 841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:13.167915 843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:13.168831 861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:13.174402 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:13.153591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:34735 (local address 127.0.0.1:40229)
0808 04:03:13.153851 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:13.153858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:13.154316 (+ 458us) server_negotiation.cc:407] Connection header received
0808 04:03:13.154495 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:13.154501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:13.154593 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:13.154726 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:13.156911 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:13.158072 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:13.160224 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:13.161021 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:13.161150 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:13.164954 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:13.164961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:13.164966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:13.165015 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:13.169227 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:13.170446 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:13.170456 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:13.170459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:13.170631 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:13.171335 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:13.171341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:13.171344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:13.171880 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:13.173531 (+ 1651us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:13.173747 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 04:03:13.173977 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:03:13.184427 861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:13.193254 861 catalog_manager.cc:1347] Generated new cluster ID: c10d3803130844e5b3fb833d1a517569
I20260808 04:03:13.193398 861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:13.226308 861 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:13.227993 861 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:13.241010 861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 796f099f29aa48d1aa15aade37ee0168: Generated new TSK 0
I20260808 04:03:13.242039 861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:13.260893 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 699
I20260808 04:03:13.262136 815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:13.358213 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 699
W20260808 04:03:13.383998 674 connection.cc:535] client connection to 127.0.0.1:40229 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40229): Connection reset by peer (error 104)
2026-08-08T04:03:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:13 dist-test-slave-pww0 krb5kdc[879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:13 dist-test-slave-pww0 krb5kdc[879](info): set up 2 sockets
Aug 08 04:03:13 dist-test-slave-pww0 krb5kdc[879](info): commencing operation
krb5kdc: starting...
W20260808 04:03:14.651597 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.003s sys 0.065s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:14 dist-test-slave-pww0 krb5kdc[879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:03:15.031005 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:15.394235 897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:15.394591 897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:15.394750 897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:15.402163 897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:15.402310 897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:15.402395 897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:15.402482 897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:15.402544 897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:15.412746 897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44805
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:15.414232 897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:15.415661 897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:15.427641 913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:15.427871 908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:15.427688 911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:15.428736 897 server_base.cc:1034] running on GCE node
I20260808 04:03:15.429598 897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:15.431129 897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:15.432343 897 hybrid_clock.cc:648] HybridClock initialized: now 1786161795432305 us; error 51 us; skew 500 ppm
I20260808 04:03:15.432691 897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:15 dist-test-slave-pww0 krb5kdc[879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:15.435792 897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:03:15.437420 897 webserver.cc:460] Webserver started at http://127.0.0.1:37115/ using document root <none> and password file <none>
I20260808 04:03:15.438104 897 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:15.438242 897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:15.438547 897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:15.441689 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "05fa8b72b82e4230a0280ade028f9962"
format_stamp: "Formatted at 2026-08-08 04:03:15 on dist-test-slave-pww0"
server_key: "cda35efecf76da8109cb2852554f13f3"
server_key_iv: "be4f72f6cbf7307013af9aace4acd660"
server_key_version: "encryptionkey@0"
I20260808 04:03:15.442495 897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "05fa8b72b82e4230a0280ade028f9962"
format_stamp: "Formatted at 2026-08-08 04:03:15 on dist-test-slave-pww0"
server_key: "cda35efecf76da8109cb2852554f13f3"
server_key_iv: "be4f72f6cbf7307013af9aace4acd660"
server_key_version: "encryptionkey@0"
I20260808 04:03:15.448047 897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 04:03:15.451891 923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:15.453327 897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:03:15.453513 897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "05fa8b72b82e4230a0280ade028f9962"
format_stamp: "Formatted at 2026-08-08 04:03:15 on dist-test-slave-pww0"
server_key: "cda35efecf76da8109cb2852554f13f3"
server_key_iv: "be4f72f6cbf7307013af9aace4acd660"
server_key_version: "encryptionkey@0"
I20260808 04:03:15.453719 897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:15.465972 897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:15.470300 897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:15.470609 897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:15.489382 897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44805
I20260808 04:03:15.489413 1012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44805 every 8 connection(s)
I20260808 04:03:15.490899 897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:15.494889 1013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:15.498464 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 897
I20260808 04:03:15.498670 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:15.499145 29679 external_mini_cluster.cc:1428] Setting key e78974d4e55cf0ab23e102787f6539d9
I20260808 04:03:15.507876 1013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962: Bootstrap starting.
I20260808 04:03:15.511900 1013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:15.513186 1013 log.cc:824] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:15.513660 1022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:15.501093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:41623 (local address 127.0.0.1:44805)
0808 04:03:15.501893 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:15.501910 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:15.501953 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:03:15.503718 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:15.503744 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:15.504205 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:15.504563 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:15.505229 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:15.506597 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:15.508518 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:15.509249 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:15.509459 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:15.510951 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:15.510960 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:15.510976 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:15.511632 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:15.511727 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:15.511760 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:15.511914 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:15.512272 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 04:03:15.512508 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":335,"threads_started":1}
I20260808 04:03:15.516937 1013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962: No bootstrap required, opened a new log
I20260808 04:03:15.518388 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:15.526021 1013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa8b72b82e4230a0280ade028f9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44805 } }
I20260808 04:03:15.526599 1013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:15.526747 1013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fa8b72b82e4230a0280ade028f9962, State: Initialized, Role: FOLLOWER
I20260808 04:03:15.527518 1013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa8b72b82e4230a0280ade028f9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44805 } }
I20260808 04:03:15.527817 1013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:15.527966 1013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:15.528138 1013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:15.530110 1022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:15.523012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:40309 (local address 127.0.0.1:44805)
0808 04:03:15.523240 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:15.523247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:15.523626 (+ 379us) server_negotiation.cc:407] Connection header received
0808 04:03:15.523853 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:15.523857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:15.523933 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:15.524030 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:15.524677 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:15.525464 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:15.527435 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:15.528013 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:15.528157 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:15.529334 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:15.529338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:15.529342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:15.529366 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:15.529428 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:15.529444 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:15.529555 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:15.529764 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 04:03:15.529835 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 04:03:15.531041 1013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa8b72b82e4230a0280ade028f9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44805 } }
I20260808 04:03:15.531700 1013 leader_election.cc:302] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05fa8b72b82e4230a0280ade028f9962; no voters:
I20260808 04:03:15.532375 1013 leader_election.cc:288] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:15.532507 1030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:15.533226 1030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [term 1 LEADER]: Becoming Leader. State: Replica: 05fa8b72b82e4230a0280ade028f9962, State: Running, Role: LEADER
I20260808 04:03:15.533797 1030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa8b72b82e4230a0280ade028f9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44805 } }
I20260808 04:03:15.534453 1013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:15.536798 1031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05fa8b72b82e4230a0280ade028f9962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa8b72b82e4230a0280ade028f9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44805 } } }
I20260808 04:03:15.537740 1031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:15.537384 1032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05fa8b72b82e4230a0280ade028f9962. Latest consensus state: current_term: 1 leader_uuid: "05fa8b72b82e4230a0280ade028f9962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa8b72b82e4230a0280ade028f9962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44805 } } }
I20260808 04:03:15.538236 1032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:15.544528 1042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:15.550108 1042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:15.559832 1042 catalog_manager.cc:1347] Generated new cluster ID: e538d877d614445aaf8629c0ab295382
I20260808 04:03:15.560036 1042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:15.585368 1042 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:15.586747 1042 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:15.596387 1042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05fa8b72b82e4230a0280ade028f9962: Generated new TSK 0
I20260808 04:03:15.597329 1042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:15.610152 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 897
W20260808 04:03:15.612011 1003 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.195:41623
W20260808 04:03:15.612862 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 897: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:15.613034 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 897
W20260808 04:03:15.638536 875 connection.cc:535] client connection to 127.0.0.1:44805 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44805): Connection reset by peer (error 104)
2026-08-08T04:03:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:15 dist-test-slave-pww0 krb5kdc[1068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:15 dist-test-slave-pww0 krb5kdc[1068](info): set up 2 sockets
Aug 08 04:03:15 dist-test-slave-pww0 krb5kdc[1068](info): commencing operation
krb5kdc: starting...
W20260808 04:03:16.901327 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.000s sys 0.068s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:17 dist-test-slave-pww0 krb5kdc[1068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:03:17.287158 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:17.655673 1093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:17.656037 1093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:17.656176 1093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:17.663831 1093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:17.663976 1093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:17.664058 1093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:17.664146 1093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:17.664207 1093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:17.674727 1093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:17.676203 1093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:17.677660 1093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:17.690536 1102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:17.690521 1100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:17.690814 1108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:17.691473 1093 server_base.cc:1034] running on GCE node
I20260808 04:03:17.692364 1093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:17.693987 1093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:17.695254 1093 hybrid_clock.cc:648] HybridClock initialized: now 1786161797695196 us; error 53 us; skew 500 ppm
I20260808 04:03:17.695768 1093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:17 dist-test-slave-pww0 krb5kdc[1068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:17.699294 1093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:03:17.701066 1093 webserver.cc:460] Webserver started at http://127.0.0.1:43199/ using document root <none> and password file <none>
I20260808 04:03:17.701872 1093 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:17.702015 1093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:17.702370 1093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:17.705890 1093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "7ff727fe1db54545aef990b72ebd5df6"
format_stamp: "Formatted at 2026-08-08 04:03:17 on dist-test-slave-pww0"
server_key: "fbf382ecc9e9d3bc4c5eb0a161eae287"
server_key_iv: "2f75822f902548daa5ba03089f70d285"
server_key_version: "encryptionkey@0"
I20260808 04:03:17.706799 1093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "7ff727fe1db54545aef990b72ebd5df6"
format_stamp: "Formatted at 2026-08-08 04:03:17 on dist-test-slave-pww0"
server_key: "fbf382ecc9e9d3bc4c5eb0a161eae287"
server_key_iv: "2f75822f902548daa5ba03089f70d285"
server_key_version: "encryptionkey@0"
I20260808 04:03:17.712908 1093 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 04:03:17.717031 1127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:17.718667 1093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:03:17.718890 1093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "7ff727fe1db54545aef990b72ebd5df6"
format_stamp: "Formatted at 2026-08-08 04:03:17 on dist-test-slave-pww0"
server_key: "fbf382ecc9e9d3bc4c5eb0a161eae287"
server_key_iv: "2f75822f902548daa5ba03089f70d285"
server_key_version: "encryptionkey@0"
I20260808 04:03:17.719115 1093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:17.725472 1093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:17.730026 1093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:17.730338 1093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:17.749269 1093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39121
I20260808 04:03:17.749336 1209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39121 every 8 connection(s)
I20260808 04:03:17.750798 1093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:17.754851 1212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:17.754987 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 1093
I20260808 04:03:17.755160 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:17.755666 29679 external_mini_cluster.cc:1428] Setting key d1d9a8c6e3c3f99666749a8b4bc0c8ad
I20260808 04:03:17.763885 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:17.757505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:53085 (local address 127.0.0.1:39121)
0808 04:03:17.758392 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:17.758406 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:17.758453 (+ 47us) server_negotiation.cc:407] Connection header received
0808 04:03:17.760103 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:17.760130 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:17.760523 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:17.760919 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:17.762074 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:17.762080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:17.762095 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:17.762115 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:17.762184 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:17.762208 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:17.762397 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:17.762510 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 04:03:17.762716 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":379,"threads_started":1}
I20260808 04:03:17.768151 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:17.769330 1212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6: Bootstrap starting.
I20260808 04:03:17.772914 1212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:17.774361 1212 log.cc:824] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:17.775483 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:17.772762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:40277 (local address 127.0.0.1:39121)
0808 04:03:17.772973 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:17.772979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:17.773389 (+ 410us) server_negotiation.cc:407] Connection header received
0808 04:03:17.773597 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:17.773601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:17.773658 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:17.773748 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:17.774720 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:17.774724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:17.774727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:17.774743 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:17.774803 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:17.774818 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:17.774917 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:17.775088 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 04:03:17.775163 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 04:03:17.777609 1212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6: No bootstrap required, opened a new log
I20260808 04:03:17.785249 1212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff727fe1db54545aef990b72ebd5df6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260808 04:03:17.785686 1212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:17.785785 1212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff727fe1db54545aef990b72ebd5df6, State: Initialized, Role: FOLLOWER
I20260808 04:03:17.786396 1212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff727fe1db54545aef990b72ebd5df6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260808 04:03:17.786640 1212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:17.786783 1212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:17.786945 1212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:17.789325 1212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff727fe1db54545aef990b72ebd5df6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260808 04:03:17.789950 1212 leader_election.cc:302] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ff727fe1db54545aef990b72ebd5df6; no voters:
I20260808 04:03:17.790593 1212 leader_election.cc:288] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:17.790735 1235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:17.791414 1235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [term 1 LEADER]: Becoming Leader. State: Replica: 7ff727fe1db54545aef990b72ebd5df6, State: Running, Role: LEADER
I20260808 04:03:17.792101 1235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff727fe1db54545aef990b72ebd5df6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260808 04:03:17.792830 1212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:17.795058 1240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ff727fe1db54545aef990b72ebd5df6. Latest consensus state: current_term: 1 leader_uuid: "7ff727fe1db54545aef990b72ebd5df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff727fe1db54545aef990b72ebd5df6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260808 04:03:17.795621 1240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:17.795109 1237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ff727fe1db54545aef990b72ebd5df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff727fe1db54545aef990b72ebd5df6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260808 04:03:17.796667 1237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:17.796725 1254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:17.802938 1254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:17.814253 1254 catalog_manager.cc:1347] Generated new cluster ID: 4829913ee95a4fcb8ae5a12bf6194ec8
I20260808 04:03:17.814419 1254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:17.830449 1254 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:17.831049 1254 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:17.844723 1254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ff727fe1db54545aef990b72ebd5df6: Generated new TSK 0
I20260808 04:03:17.845669 1254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:17.861897 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1093
W20260808 04:03:17.863101 1205 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.195:53085
W20260808 04:03:17.863909 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1093: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:17.864060 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 1093
2026-08-08T04:03:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:17 dist-test-slave-pww0 krb5kdc[1272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:17 dist-test-slave-pww0 krb5kdc[1272](info): set up 2 sockets
Aug 08 04:03:17 dist-test-slave-pww0 krb5kdc[1272](info): commencing operation
krb5kdc: starting...
W20260808 04:03:19.157912 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.003s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:19 dist-test-slave-pww0 krb5kdc[1272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:03:19.534471 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41355
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:19.900271 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:19.900615 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:19.900771 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:19.908362 1291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:19.908516 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:19.908608 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:19.908702 1291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:19.908761 1291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:19.919263 1291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:41355
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:19.920747 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:19.922075 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:19.933938 1308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:19.933992 1303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:19.934235 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:19.934911 1291 server_base.cc:1034] running on GCE node
I20260808 04:03:19.935719 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:19.937160 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:19.938378 1291 hybrid_clock.cc:648] HybridClock initialized: now 1786161799938322 us; error 48 us; skew 500 ppm
I20260808 04:03:19.938742 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:19 dist-test-slave-pww0 krb5kdc[1272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161799, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:19.946856 1291 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:03:19.948755 1291 webserver.cc:460] Webserver started at http://127.28.251.254:39293/ using document root <none> and password file <none>
I20260808 04:03:19.949549 1291 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:19.949695 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:19.950018 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:19.953439 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "59553795671b40b485e1d8e0ab175145"
format_stamp: "Formatted at 2026-08-08 04:03:19 on dist-test-slave-pww0"
server_key: "69684ff7f81e9fc6f5ec08a9de2213b1"
server_key_iv: "7b589829326534aca1e8e90ad34f8fec"
server_key_version: "encryptionkey@0"
I20260808 04:03:19.954341 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "59553795671b40b485e1d8e0ab175145"
format_stamp: "Formatted at 2026-08-08 04:03:19 on dist-test-slave-pww0"
server_key: "69684ff7f81e9fc6f5ec08a9de2213b1"
server_key_iv: "7b589829326534aca1e8e90ad34f8fec"
server_key_version: "encryptionkey@0"
I20260808 04:03:19.960208 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260808 04:03:19.963974 1319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:19.965590 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 04:03:19.965816 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "59553795671b40b485e1d8e0ab175145"
format_stamp: "Formatted at 2026-08-08 04:03:19 on dist-test-slave-pww0"
server_key: "69684ff7f81e9fc6f5ec08a9de2213b1"
server_key_iv: "7b589829326534aca1e8e90ad34f8fec"
server_key_version: "encryptionkey@0"
I20260808 04:03:19.966050 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:20.004268 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:20.009217 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:20.009536 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:20.028605 1291 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:41355
I20260808 04:03:20.028643 1388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:41355 every 8 connection(s)
I20260808 04:03:20.030049 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:20.033041 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 1291
I20260808 04:03:20.033245 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:20.033852 29679 external_mini_cluster.cc:1428] Setting key 434265ddd234b5ecdfc62283f408399b
I20260808 04:03:20.034557 1391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 04:03:20 dist-test-slave-pww0 krb5kdc[1272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:03:20.050231 1391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145: Bootstrap starting.
I20260808 04:03:20.053834 1391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:20.055246 1391 log.cc:824] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:20.058926 1391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145: No bootstrap required, opened a new log
I20260808 04:03:20.059396 1395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:20.035850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45727 (local address 127.28.251.254:41355)
0808 04:03:20.036655 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:20.036667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:20.036711 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:03:20.038289 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:20.038314 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:20.038763 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:20.039145 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:20.041265 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:20.042754 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:20.044623 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:20.045346 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:20.045502 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:20.050861 (+ 5359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:20.050871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:20.050888 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:20.050935 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:20.054092 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:20.055197 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:20.055204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:20.055211 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:20.055397 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:20.056013 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:20.056018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:20.056025 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:20.056868 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:20.057139 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:20.057824 (+ 685us) server_negotiation.cc:299] Negotiation successful
0808 04:03:20.058214 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":350,"threads_started":1}
I20260808 04:03:20.063877 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:20.067806 1391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59553795671b40b485e1d8e0ab175145" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41355 } }
I20260808 04:03:20.068356 1391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:20.068521 1391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59553795671b40b485e1d8e0ab175145, State: Initialized, Role: FOLLOWER
I20260808 04:03:20.069346 1391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59553795671b40b485e1d8e0ab175145" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41355 } }
I20260808 04:03:20.069631 1391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:20.069880 1391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:20.070122 1391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:20.073204 1391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59553795671b40b485e1d8e0ab175145" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41355 } }
I20260808 04:03:20.073889 1391 leader_election.cc:302] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59553795671b40b485e1d8e0ab175145; no voters:
I20260808 04:03:20.074653 1391 leader_election.cc:288] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:20.074771 1412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:20.076691 1412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [term 1 LEADER]: Becoming Leader. State: Replica: 59553795671b40b485e1d8e0ab175145, State: Running, Role: LEADER
I20260808 04:03:20.077306 1391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:20.077442 1412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59553795671b40b485e1d8e0ab175145" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41355 } }
I20260808 04:03:20.080673 1413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59553795671b40b485e1d8e0ab175145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59553795671b40b485e1d8e0ab175145" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41355 } } }
I20260808 04:03:20.081133 1413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:20.085839 1414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59553795671b40b485e1d8e0ab175145. Latest consensus state: current_term: 1 leader_uuid: "59553795671b40b485e1d8e0ab175145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59553795671b40b485e1d8e0ab175145" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41355 } } }
I20260808 04:03:20.087179 1414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:20.088214 1395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:20.068582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41197 (local address 127.28.251.254:41355)
0808 04:03:20.068819 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:20.068826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:20.069309 (+ 483us) server_negotiation.cc:407] Connection header received
0808 04:03:20.069512 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:20.069517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:20.069587 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:20.069687 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:20.071771 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:20.072583 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:20.074331 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:20.074963 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:20.075786 (+ 823us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:20.078459 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:20.078465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:20.078468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:20.078508 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:20.082645 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:20.084154 (+ 1509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:20.084161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:20.084166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:20.084410 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:20.085189 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:20.085195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:20.085200 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:20.085694 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:20.087241 (+ 1547us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:20.087481 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 04:03:20.087707 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:03:20.092129 1420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:20.097790 1420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:20.107998 1420 catalog_manager.cc:1347] Generated new cluster ID: 512cd0b982e042ffb715e94dd60fe809
I20260808 04:03:20.108232 1420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:20.131314 1420 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:20.132926 1420 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:20.145069 1420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59553795671b40b485e1d8e0ab175145: Generated new TSK 0
I20260808 04:03:20.146045 1420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:20.152467 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1291
I20260808 04:03:20.153760 1381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:20.248159 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 1291
W20260808 04:03:20.276075 1269 connection.cc:535] client connection to 127.28.251.254:41355 recv error: Network error: failed to read from TLS socket (remote: 127.28.251.254:41355): Connection reset by peer (error 104)
2026-08-08T04:03:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:20 dist-test-slave-pww0 krb5kdc[1447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:20 dist-test-slave-pww0 krb5kdc[1447](info): set up 2 sockets
Aug 08 04:03:20 dist-test-slave-pww0 krb5kdc[1447](info): commencing operation
krb5kdc: starting...
W20260808 04:03:21.532534 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.003s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:21 dist-test-slave-pww0 krb5kdc[1447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:03:21.903962 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:36773
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:36773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:22.274374 1471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:22.274703 1471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:22.274827 1471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:22.282213 1471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:22.282341 1471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:22.282394 1471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:22.282481 1471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:22.282532 1471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:22.292693 1471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:36773
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:36773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:22.294129 1471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:22.295542 1471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:22.307511 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:22.307487 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:22.307403 1482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:22.308441 1471 server_base.cc:1034] running on GCE node
I20260808 04:03:22.309288 1471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:22.310961 1471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:22.312192 1471 hybrid_clock.cc:648] HybridClock initialized: now 1786161802312151 us; error 38 us; skew 500 ppm
I20260808 04:03:22.312602 1471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:22 dist-test-slave-pww0 krb5kdc[1447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:22.315851 1471 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:03:22.317412 1471 webserver.cc:460] Webserver started at http://127.28.251.254:35287/ using document root <none> and password file <none>
I20260808 04:03:22.318131 1471 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:22.318264 1471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:22.318568 1471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:22.321805 1471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "1a379fcb3f6940c481947ed7cf4b03c9"
format_stamp: "Formatted at 2026-08-08 04:03:22 on dist-test-slave-pww0"
server_key: "9bf0e6bf4b5eef616963ca348d34ec38"
server_key_iv: "c2316498d2914e9f43754191a2ed1232"
server_key_version: "encryptionkey@0"
I20260808 04:03:22.322647 1471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "1a379fcb3f6940c481947ed7cf4b03c9"
format_stamp: "Formatted at 2026-08-08 04:03:22 on dist-test-slave-pww0"
server_key: "9bf0e6bf4b5eef616963ca348d34ec38"
server_key_iv: "c2316498d2914e9f43754191a2ed1232"
server_key_version: "encryptionkey@0"
I20260808 04:03:22.328568 1471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260808 04:03:22.332650 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:22.334254 1471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260808 04:03:22.334467 1471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "1a379fcb3f6940c481947ed7cf4b03c9"
format_stamp: "Formatted at 2026-08-08 04:03:22 on dist-test-slave-pww0"
server_key: "9bf0e6bf4b5eef616963ca348d34ec38"
server_key_iv: "c2316498d2914e9f43754191a2ed1232"
server_key_version: "encryptionkey@0"
I20260808 04:03:22.334686 1471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:22.353497 1471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:22.358171 1471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:22.358479 1471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:22.376904 1471 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:36773
I20260808 04:03:22.376915 1576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:36773 every 8 connection(s)
I20260808 04:03:22.378335 1471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:22.381788 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 1471
I20260808 04:03:22.381992 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:22.382576 29679 external_mini_cluster.cc:1428] Setting key b1dacc956174c54b4349e01ea71ec612
I20260808 04:03:22.382683 1577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:22.397166 1577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9: Bootstrap starting.
I20260808 04:03:22.397199 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:22.384462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43785 (local address 127.28.251.254:36773)
0808 04:03:22.385264 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:22.385280 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:22.385324 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:03:22.387438 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:22.387496 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:22.387984 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:22.388341 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:22.389069 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:22.390462 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:22.392397 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:22.393157 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:22.393317 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:22.394706 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:22.394715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:22.394731 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:22.395366 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:22.395438 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:22.395509 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:22.395646 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:22.395933 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 04:03:22.396133 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":368,"threads_started":1}
I20260808 04:03:22.400779 1577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:22.401345 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:22.402101 1577 log.cc:824] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:22.405782 1577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9: No bootstrap required, opened a new log
I20260808 04:03:22.414806 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:22.406540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56611 (local address 127.28.251.254:36773)
0808 04:03:22.406746 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:22.406753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:22.408061 (+ 1308us) server_negotiation.cc:407] Connection header received
0808 04:03:22.408263 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:22.408267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:22.408349 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:22.408459 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:22.409099 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:22.409908 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:22.411934 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:22.412530 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:22.412645 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:22.413969 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:22.413975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:22.413981 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:22.414007 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:22.414071 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:22.414089 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:22.414223 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:22.414401 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 04:03:22.414455 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:03:22.416002 1577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a379fcb3f6940c481947ed7cf4b03c9" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36773 } }
I20260808 04:03:22.416494 1577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:22.416599 1577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a379fcb3f6940c481947ed7cf4b03c9, State: Initialized, Role: FOLLOWER
I20260808 04:03:22.417233 1577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a379fcb3f6940c481947ed7cf4b03c9" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36773 } }
I20260808 04:03:22.417483 1577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:22.417616 1577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:22.417779 1577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:22.420337 1577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a379fcb3f6940c481947ed7cf4b03c9" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36773 } }
I20260808 04:03:22.420897 1577 leader_election.cc:302] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a379fcb3f6940c481947ed7cf4b03c9; no voters:
I20260808 04:03:22.421537 1577 leader_election.cc:288] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:22.421703 1598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:22.422443 1598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [term 1 LEADER]: Becoming Leader. State: Replica: 1a379fcb3f6940c481947ed7cf4b03c9, State: Running, Role: LEADER
I20260808 04:03:22.423080 1598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a379fcb3f6940c481947ed7cf4b03c9" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36773 } }
I20260808 04:03:22.423794 1577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:22.426343 1601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a379fcb3f6940c481947ed7cf4b03c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a379fcb3f6940c481947ed7cf4b03c9" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36773 } } }
I20260808 04:03:22.426818 1601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:22.426316 1602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a379fcb3f6940c481947ed7cf4b03c9. Latest consensus state: current_term: 1 leader_uuid: "1a379fcb3f6940c481947ed7cf4b03c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a379fcb3f6940c481947ed7cf4b03c9" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 36773 } } }
I20260808 04:03:22.427380 1602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:22.430583 1609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:22.436805 1609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:22.447893 1609 catalog_manager.cc:1347] Generated new cluster ID: 2f00b26926b94409bf5886d0db460891
I20260808 04:03:22.448031 1609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:22.472600 1609 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:22.474579 1609 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:22.491199 1609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a379fcb3f6940c481947ed7cf4b03c9: Generated new TSK 0
I20260808 04:03:22.492139 1609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:22.505818 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1471
W20260808 04:03:22.507208 1569 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43785
W20260808 04:03:22.508067 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1471: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:22.508222 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 1471
W20260808 04:03:22.537395 1445 connection.cc:535] client connection to 127.28.251.254:36773 recv error: Network error: failed to read from TLS socket (remote: 127.28.251.254:36773): Connection reset by peer (error 104)
2026-08-08T04:03:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2261 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:22 dist-test-slave-pww0 krb5kdc[1638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:22 dist-test-slave-pww0 krb5kdc[1638](info): set up 2 sockets
Aug 08 04:03:22 dist-test-slave-pww0 krb5kdc[1638](info): commencing operation
krb5kdc: starting...
W20260808 04:03:23.805639 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:24 dist-test-slave-pww0 krb5kdc[1638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:03:24.187674 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:34349
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:34349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:24.553947 1656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:24.554263 1656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:24.554395 1656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:24.561960 1656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:24.562104 1656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:24.562165 1656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:24.562255 1656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:24.562309 1656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:24.572876 1656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:34349
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:34349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:24.574282 1656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:24.575748 1656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:24.588459 1663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:24.588577 1662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:24.589001 1666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:24.589257 1656 server_base.cc:1034] running on GCE node
I20260808 04:03:24.590116 1656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:24.591710 1656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:24.592922 1656 hybrid_clock.cc:648] HybridClock initialized: now 1786161804592861 us; error 61 us; skew 500 ppm
I20260808 04:03:24.593288 1656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:24 dist-test-slave-pww0 krb5kdc[1638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161804, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:24.596621 1656 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:03:24.598251 1656 webserver.cc:460] Webserver started at http://127.28.251.254:34369/ using document root <none> and password file <none>
I20260808 04:03:24.598958 1656 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:24.599062 1656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:24.599346 1656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:24.602568 1656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "bc741fcbb2f54d9eae119f248dd4faec"
format_stamp: "Formatted at 2026-08-08 04:03:24 on dist-test-slave-pww0"
server_key: "2285cf8f816eb8bc93ff8a4b3058265a"
server_key_iv: "1d66b5de6f14f8163aac1d64fa314695"
server_key_version: "encryptionkey@0"
I20260808 04:03:24.603374 1656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "bc741fcbb2f54d9eae119f248dd4faec"
format_stamp: "Formatted at 2026-08-08 04:03:24 on dist-test-slave-pww0"
server_key: "2285cf8f816eb8bc93ff8a4b3058265a"
server_key_iv: "1d66b5de6f14f8163aac1d64fa314695"
server_key_version: "encryptionkey@0"
I20260808 04:03:24.608965 1656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260808 04:03:24.612982 1675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:24.614658 1656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:03:24.614861 1656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "bc741fcbb2f54d9eae119f248dd4faec"
format_stamp: "Formatted at 2026-08-08 04:03:24 on dist-test-slave-pww0"
server_key: "2285cf8f816eb8bc93ff8a4b3058265a"
server_key_iv: "1d66b5de6f14f8163aac1d64fa314695"
server_key_version: "encryptionkey@0"
I20260808 04:03:24.615089 1656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:24.621446 1656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:24.626075 1656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:24.626384 1656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:24.645188 1656 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:34349
I20260808 04:03:24.645221 1752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:34349 every 8 connection(s)
I20260808 04:03:24.646730 1656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:24.650902 1753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:24.656265 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 1656
I20260808 04:03:24.656481 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:24.657047 29679 external_mini_cluster.cc:1428] Setting key 08afe5a5ab449296b9d5a0611a720c70
I20260808 04:03:24.663587 1753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec: Bootstrap starting.
I20260808 04:03:24.665665 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:24.659133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44037 (local address 127.28.251.254:34349)
0808 04:03:24.660027 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:24.660039 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:24.660087 (+ 48us) server_negotiation.cc:407] Connection header received
0808 04:03:24.661966 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:24.661995 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:24.662378 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:24.662753 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:24.663958 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:24.663965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:24.663979 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:24.664003 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:24.664068 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:24.664099 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:24.664294 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:24.664405 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 04:03:24.664589 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":388,"threads_started":1}
I20260808 04:03:24.667335 1753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:24.668604 1753 log.cc:824] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:24.669898 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:24.671622 1753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec: No bootstrap required, opened a new log
I20260808 04:03:24.677440 1760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:24.674711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33303 (local address 127.28.251.254:34349)
0808 04:03:24.674946 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:24.674953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:24.675261 (+ 308us) server_negotiation.cc:407] Connection header received
0808 04:03:24.675523 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:24.675527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:24.675593 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:24.675720 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:24.676713 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:24.676717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:24.676720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:24.676737 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:24.676786 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:24.676810 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:24.676924 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:24.677040 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 04:03:24.677101 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260808 04:03:24.680357 1753 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc741fcbb2f54d9eae119f248dd4faec" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34349 } }
I20260808 04:03:24.680768 1753 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:24.680867 1753 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc741fcbb2f54d9eae119f248dd4faec, State: Initialized, Role: FOLLOWER
I20260808 04:03:24.681430 1753 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc741fcbb2f54d9eae119f248dd4faec" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34349 } }
I20260808 04:03:24.681699 1753 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:24.681828 1753 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:24.681983 1753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:24.684532 1753 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc741fcbb2f54d9eae119f248dd4faec" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34349 } }
I20260808 04:03:24.685077 1753 leader_election.cc:302] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc741fcbb2f54d9eae119f248dd4faec; no voters:
I20260808 04:03:24.685683 1753 leader_election.cc:288] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:24.685855 1768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:24.686535 1768 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [term 1 LEADER]: Becoming Leader. State: Replica: bc741fcbb2f54d9eae119f248dd4faec, State: Running, Role: LEADER
I20260808 04:03:24.687096 1768 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc741fcbb2f54d9eae119f248dd4faec" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34349 } }
I20260808 04:03:24.687841 1753 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:24.690367 1770 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc741fcbb2f54d9eae119f248dd4faec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc741fcbb2f54d9eae119f248dd4faec" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34349 } } }
I20260808 04:03:24.690793 1770 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:24.692090 1778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:24.692013 1773 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc741fcbb2f54d9eae119f248dd4faec. Latest consensus state: current_term: 1 leader_uuid: "bc741fcbb2f54d9eae119f248dd4faec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc741fcbb2f54d9eae119f248dd4faec" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 34349 } } }
I20260808 04:03:24.692340 1773 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:24.697481 1778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:24.708627 1778 catalog_manager.cc:1347] Generated new cluster ID: 7483cd7a444c4d99a61ddb5b8f82bb0f
I20260808 04:03:24.708775 1778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:24.727054 1778 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:24.727636 1778 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:24.738222 1778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc741fcbb2f54d9eae119f248dd4faec: Generated new TSK 0
I20260808 04:03:24.739300 1778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:24.756095 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1656
W20260808 04:03:24.757426 1748 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44037
W20260808 04:03:24.758271 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1656: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:24.758430 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 1656
2026-08-08T04:03:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:24 dist-test-slave-pww0 krb5kdc[1803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:24 dist-test-slave-pww0 krb5kdc[1803](info): set up 2 sockets
Aug 08 04:03:24 dist-test-slave-pww0 krb5kdc[1803](info): commencing operation
krb5kdc: starting...
W20260808 04:03:26.064973 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.002s sys 0.063s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:26 dist-test-slave-pww0 krb5kdc[1803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:03:26.450719 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41003
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:41003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:26.818914 1825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:26.819250 1825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:26.819380 1825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:26.826949 1825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:26.827081 1825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:26.827150 1825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:26.827236 1825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:26.827286 1825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:26.837787 1825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:41003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:26.839215 1825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:26.840663 1825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:26.853386 1834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:26.853345 1835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:26.853307 1838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:26.854117 1825 server_base.cc:1034] running on GCE node
I20260808 04:03:26.854987 1825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:26.856581 1825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:26.857812 1825 hybrid_clock.cc:648] HybridClock initialized: now 1786161806857740 us; error 62 us; skew 500 ppm
I20260808 04:03:26.858227 1825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:26 dist-test-slave-pww0 krb5kdc[1803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161806, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:26.866154 1825 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:03:26.868186 1825 webserver.cc:460] Webserver started at http://127.28.251.254:45359/ using document root <none> and password file <none>
I20260808 04:03:26.869001 1825 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:26.869169 1825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:26.869537 1825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:26.873167 1825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "6205c9216f8c4886835dcfa3d0ce25b1"
format_stamp: "Formatted at 2026-08-08 04:03:26 on dist-test-slave-pww0"
server_key: "48edce2ca37fcf73a6a662c04b778aaa"
server_key_iv: "3c2a4f10dc4d2ea29a41eb36ce6b9b90"
server_key_version: "encryptionkey@0"
I20260808 04:03:26.874121 1825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "6205c9216f8c4886835dcfa3d0ce25b1"
format_stamp: "Formatted at 2026-08-08 04:03:26 on dist-test-slave-pww0"
server_key: "48edce2ca37fcf73a6a662c04b778aaa"
server_key_iv: "3c2a4f10dc4d2ea29a41eb36ce6b9b90"
server_key_version: "encryptionkey@0"
I20260808 04:03:26.880494 1825 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260808 04:03:26.884717 1849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:26.886168 1825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260808 04:03:26.886356 1825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "6205c9216f8c4886835dcfa3d0ce25b1"
format_stamp: "Formatted at 2026-08-08 04:03:26 on dist-test-slave-pww0"
server_key: "48edce2ca37fcf73a6a662c04b778aaa"
server_key_iv: "3c2a4f10dc4d2ea29a41eb36ce6b9b90"
server_key_version: "encryptionkey@0"
I20260808 04:03:26.886574 1825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:26.909469 1825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:26.914196 1825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:26.914484 1825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:26.935519 1825 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:41003
I20260808 04:03:26.935621 1929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:41003 every 8 connection(s)
I20260808 04:03:26.936980 1825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:26.940018 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 1825
I20260808 04:03:26.940218 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:26.940797 29679 external_mini_cluster.cc:1428] Setting key 62c7e4068955e5598c8c48ea615da080
I20260808 04:03:26.941376 1931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:26.956890 1931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1: Bootstrap starting.
Aug 08 04:03:26 dist-test-slave-pww0 krb5kdc[1803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:03:26.960382 1931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:26.961829 1931 log.cc:824] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:26.965540 1931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1: No bootstrap required, opened a new log
I20260808 04:03:26.967787 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:26.942830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48773 (local address 127.28.251.254:41003)
0808 04:03:26.943760 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:26.943774 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:26.943826 (+ 52us) server_negotiation.cc:407] Connection header received
0808 04:03:26.946525 (+ 2699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:26.946555 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:26.947058 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:26.947541 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:26.949552 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:26.950997 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:26.952931 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:26.953731 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:26.953912 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:26.958803 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:26.958812 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:26.958827 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:26.958882 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:26.962566 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:26.963560 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:26.963569 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:26.963585 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:26.963839 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:26.964239 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:26.964245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:26.964248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:26.965006 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:26.965295 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:26.965824 (+ 529us) server_negotiation.cc:299] Negotiation successful
0808 04:03:26.966302 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":405,"threads_started":1}
I20260808 04:03:26.972106 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:26.974884 1931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205c9216f8c4886835dcfa3d0ce25b1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } }
I20260808 04:03:26.975395 1931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:26.975558 1931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6205c9216f8c4886835dcfa3d0ce25b1, State: Initialized, Role: FOLLOWER
I20260808 04:03:26.976251 1931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205c9216f8c4886835dcfa3d0ce25b1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } }
I20260808 04:03:26.976504 1931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:26.976660 1931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:26.976876 1931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:26.979614 1931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205c9216f8c4886835dcfa3d0ce25b1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } }
I20260808 04:03:26.980386 1931 leader_election.cc:302] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6205c9216f8c4886835dcfa3d0ce25b1; no voters:
I20260808 04:03:26.982414 1931 leader_election.cc:288] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:26.982527 1943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:26.984179 1943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [term 1 LEADER]: Becoming Leader. State: Replica: 6205c9216f8c4886835dcfa3d0ce25b1, State: Running, Role: LEADER
I20260808 04:03:26.984766 1943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205c9216f8c4886835dcfa3d0ce25b1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } }
I20260808 04:03:26.985098 1931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:26.987576 1948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6205c9216f8c4886835dcfa3d0ce25b1. Latest consensus state: current_term: 1 leader_uuid: "6205c9216f8c4886835dcfa3d0ce25b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205c9216f8c4886835dcfa3d0ce25b1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } } }
I20260808 04:03:26.987867 1944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6205c9216f8c4886835dcfa3d0ce25b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205c9216f8c4886835dcfa3d0ce25b1" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41003 } } }
I20260808 04:03:26.988057 1948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:26.988157 1944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:26.994675 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:26.976870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37697 (local address 127.28.251.254:41003)
0808 04:03:26.977108 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:26.977116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:26.977736 (+ 620us) server_negotiation.cc:407] Connection header received
0808 04:03:26.977901 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:26.977907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:26.978000 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:26.978115 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:26.979678 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:26.980732 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:26.982129 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:26.982850 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:26.983005 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:26.986533 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:26.986542 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:26.986548 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:26.986605 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:26.990941 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:26.992114 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:26.992121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:26.992125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:26.992309 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:26.992772 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:26.992777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:26.992781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:26.993334 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:26.993650 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:26.994012 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 04:03:26.994271 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:03:26.997352 1957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:27.003594 1957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:27.014708 1957 catalog_manager.cc:1347] Generated new cluster ID: d1f0ec52bc1e4117be39818153849a1b
I20260808 04:03:27.014861 1957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:27.048413 1957 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:27.049890 1957 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:27.060490 1957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6205c9216f8c4886835dcfa3d0ce25b1: Generated new TSK 0
I20260808 04:03:27.061477 1957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:27.071072 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 1825
I20260808 04:03:27.072492 1925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:27.171288 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 1825
2026-08-08T04:03:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:27 dist-test-slave-pww0 krb5kdc[1977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:27 dist-test-slave-pww0 krb5kdc[1977](info): set up 2 sockets
Aug 08 04:03:27 dist-test-slave-pww0 krb5kdc[1977](info): commencing operation
krb5kdc: starting...
W20260808 04:03:28.472390 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.000s sys 0.065s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:28 dist-test-slave-pww0 krb5kdc[1977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:03:28.851148 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41269
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:44971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:41269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:29.221724 2006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:29.222054 2006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:29.222206 2006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:29.229797 2006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:29.229941 2006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:29.230005 2006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:29.230093 2006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:29.230161 2006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:29.240756 2006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:44971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:41269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:41269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:29.242174 2006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:29.243597 2006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:29.255829 2020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:29.255822 2015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:29.255821 2014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:29.256937 2006 server_base.cc:1034] running on GCE node
I20260808 04:03:29.257828 2006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:29.259522 2006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:29.260778 2006 hybrid_clock.cc:648] HybridClock initialized: now 1786161809260730 us; error 78 us; skew 500 ppm
I20260808 04:03:29.261191 2006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:29 dist-test-slave-pww0 krb5kdc[1977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161809, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:29.264614 2006 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:03:29.266319 2006 webserver.cc:460] Webserver started at http://127.28.251.254:38167/ using document root <none> and password file <none>
I20260808 04:03:29.267057 2006 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:29.267192 2006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:29.267549 2006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:29.270818 2006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "ba8cfd93b4fc4a90bc1f19420fa92542"
format_stamp: "Formatted at 2026-08-08 04:03:29 on dist-test-slave-pww0"
server_key: "5db6ef7e7f6405445e29a1c8de61a633"
server_key_iv: "8bc5d0d06336e3a1f8f8ebb1c719acac"
server_key_version: "encryptionkey@0"
I20260808 04:03:29.271679 2006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "ba8cfd93b4fc4a90bc1f19420fa92542"
format_stamp: "Formatted at 2026-08-08 04:03:29 on dist-test-slave-pww0"
server_key: "5db6ef7e7f6405445e29a1c8de61a633"
server_key_iv: "8bc5d0d06336e3a1f8f8ebb1c719acac"
server_key_version: "encryptionkey@0"
I20260808 04:03:29.277381 2006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:03:29.281217 2030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:29.282706 2006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260808 04:03:29.282900 2006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "ba8cfd93b4fc4a90bc1f19420fa92542"
format_stamp: "Formatted at 2026-08-08 04:03:29 on dist-test-slave-pww0"
server_key: "5db6ef7e7f6405445e29a1c8de61a633"
server_key_iv: "8bc5d0d06336e3a1f8f8ebb1c719acac"
server_key_version: "encryptionkey@0"
I20260808 04:03:29.283108 2006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:29.317174 2006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:29.322067 2006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:29.322371 2006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:29.341329 2006 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:41269
I20260808 04:03:29.341370 2110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:41269 every 8 connection(s)
I20260808 04:03:29.342846 2006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:29.346953 2111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:29.350167 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 2006
I20260808 04:03:29.350369 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:29.350795 29679 external_mini_cluster.cc:1428] Setting key 779cc554554e2f6e74038be2f44b8c19
I20260808 04:03:29.359774 2111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542: Bootstrap starting.
I20260808 04:03:29.363725 2111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:29.364986 2111 log.cc:824] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:29.365123 2116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:29.352743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51955 (local address 127.28.251.254:41269)
0808 04:03:29.353590 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:29.353603 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:29.353650 (+ 47us) server_negotiation.cc:407] Connection header received
0808 04:03:29.355447 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:29.355511 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:29.356002 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:29.356384 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:29.357087 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:29.358448 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:29.360355 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:29.361087 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:29.361230 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:29.362607 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:29.362615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:29.362632 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:29.363205 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:29.363294 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:29.363321 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:29.363413 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:29.363697 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 04:03:29.364010 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":397,"threads_started":1}
I20260808 04:03:29.368259 2111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542: No bootstrap required, opened a new log
I20260808 04:03:29.369161 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:29.377151 2111 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8cfd93b4fc4a90bc1f19420fa92542" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41269 } }
I20260808 04:03:29.377727 2111 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:29.377858 2111 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8cfd93b4fc4a90bc1f19420fa92542, State: Initialized, Role: FOLLOWER
I20260808 04:03:29.378566 2111 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8cfd93b4fc4a90bc1f19420fa92542" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41269 } }
I20260808 04:03:29.378887 2111 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:29.379043 2111 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:29.379233 2111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:29.380753 2116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:29.373812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54947 (local address 127.28.251.254:41269)
0808 04:03:29.374044 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:29.374050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:29.374374 (+ 324us) server_negotiation.cc:407] Connection header received
0808 04:03:29.374562 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:29.374566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:29.374638 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:29.374729 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:29.375379 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:29.376195 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:29.378082 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:29.378549 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:29.378670 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:29.380004 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:29.380010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:29.380013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:29.380037 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:29.380092 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:29.380109 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:29.380189 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:29.380337 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 04:03:29.380430 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:03:29.382129 2111 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8cfd93b4fc4a90bc1f19420fa92542" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41269 } }
I20260808 04:03:29.382691 2111 leader_election.cc:302] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba8cfd93b4fc4a90bc1f19420fa92542; no voters:
I20260808 04:03:29.383294 2111 leader_election.cc:288] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:29.383431 2125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:29.384140 2125 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [term 1 LEADER]: Becoming Leader. State: Replica: ba8cfd93b4fc4a90bc1f19420fa92542, State: Running, Role: LEADER
I20260808 04:03:29.384794 2125 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8cfd93b4fc4a90bc1f19420fa92542" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41269 } }
I20260808 04:03:29.386948 2125 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba8cfd93b4fc4a90bc1f19420fa92542. Latest consensus state: current_term: 1 leader_uuid: "ba8cfd93b4fc4a90bc1f19420fa92542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8cfd93b4fc4a90bc1f19420fa92542" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41269 } } }
I20260808 04:03:29.387333 2125 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:29.387995 2132 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba8cfd93b4fc4a90bc1f19420fa92542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8cfd93b4fc4a90bc1f19420fa92542" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 41269 } } }
I20260808 04:03:29.388388 2132 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:29.392058 2111 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:29.396107 2135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:29.401047 2135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:29.411427 2135 catalog_manager.cc:1347] Generated new cluster ID: 9887580ea87b468abb766311b31dc9ac
I20260808 04:03:29.411605 2135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:29.424060 2135 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:29.425412 2135 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:29.444319 2135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba8cfd93b4fc4a90bc1f19420fa92542: Generated new TSK 0
I20260808 04:03:29.445564 2135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:29.459826 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2006
W20260808 04:03:29.461304 2102 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51955
W20260808 04:03:29.462098 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2006: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:29.462240 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 2006
2026-08-08T04:03:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:29 dist-test-slave-pww0 krb5kdc[2164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:29 dist-test-slave-pww0 krb5kdc[2164](info): set up 2 sockets
Aug 08 04:03:29 dist-test-slave-pww0 krb5kdc[2164](info): commencing operation
krb5kdc: starting...
W20260808 04:03:30.761162 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.000s sys 0.065s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:30 dist-test-slave-pww0 krb5kdc[2164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:03:31.147107 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:45353
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:45353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:31.516443 2188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:31.516777 2188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:31.516884 2188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:31.524312 2188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:31.524443 2188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:31.524502 2188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:31.524590 2188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:31.524645 2188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:31.535020 2188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:45353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:45353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:31.536481 2188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:31.537882 2188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:31.550486 2204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:31.550539 2202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:31.550573 2201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:31.551249 2188 server_base.cc:1034] running on GCE node
I20260808 04:03:31.552158 2188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:31.553799 2188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:31.555012 2188 hybrid_clock.cc:648] HybridClock initialized: now 1786161811554965 us; error 36 us; skew 500 ppm
I20260808 04:03:31.555397 2188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:31 dist-test-slave-pww0 krb5kdc[2164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161811, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:31.558640 2188 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:03:31.560361 2188 webserver.cc:460] Webserver started at http://127.28.251.254:43245/ using document root <none> and password file <none>
I20260808 04:03:31.561113 2188 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:31.561241 2188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:31.561563 2188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:31.564849 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "74744bdd2ae04a56bafe275cc204f6d8"
format_stamp: "Formatted at 2026-08-08 04:03:31 on dist-test-slave-pww0"
server_key: "f8f9568a96f8e94c26861b942979dc84"
server_key_iv: "58e9964a6092e45d1b09ee6f322d3f00"
server_key_version: "encryptionkey@0"
I20260808 04:03:31.565667 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "74744bdd2ae04a56bafe275cc204f6d8"
format_stamp: "Formatted at 2026-08-08 04:03:31 on dist-test-slave-pww0"
server_key: "f8f9568a96f8e94c26861b942979dc84"
server_key_iv: "58e9964a6092e45d1b09ee6f322d3f00"
server_key_version: "encryptionkey@0"
I20260808 04:03:31.571251 2188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 04:03:31.575186 2213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:31.576745 2188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:03:31.576953 2188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "74744bdd2ae04a56bafe275cc204f6d8"
format_stamp: "Formatted at 2026-08-08 04:03:31 on dist-test-slave-pww0"
server_key: "f8f9568a96f8e94c26861b942979dc84"
server_key_iv: "58e9964a6092e45d1b09ee6f322d3f00"
server_key_version: "encryptionkey@0"
I20260808 04:03:31.577167 2188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:31.583400 2188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:31.587920 2188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:31.588224 2188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:31.607000 2188 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:45353
I20260808 04:03:31.607043 2309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:45353 every 8 connection(s)
I20260808 04:03:31.608584 2188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:31.612591 2310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:31.616423 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 2188
I20260808 04:03:31.616587 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:31.617038 29679 external_mini_cluster.cc:1428] Setting key d2d37ca0bcd2c3660cac31be0353f6ae
I20260808 04:03:31.626057 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:31.618931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40051 (local address 127.28.251.254:45353)
0808 04:03:31.619790 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:31.619805 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:31.619859 (+ 54us) server_negotiation.cc:407] Connection header received
0808 04:03:31.621807 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:31.621843 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:31.622387 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:31.622882 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:31.624001 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:31.624010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:31.624027 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:31.624045 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:31.624119 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:31.624158 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:31.624401 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:31.624532 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 04:03:31.624745 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":332,"threads_started":1}
I20260808 04:03:31.626688 2310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8: Bootstrap starting.
I20260808 04:03:31.629935 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:31.630103 2310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:31.631441 2310 log.cc:824] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:31.635066 2310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8: No bootstrap required, opened a new log
I20260808 04:03:31.637979 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:31.635199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38003 (local address 127.28.251.254:45353)
0808 04:03:31.635376 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:31.635383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:31.635984 (+ 601us) server_negotiation.cc:407] Connection header received
0808 04:03:31.636174 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:31.636178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:31.636231 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:31.636316 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:31.637284 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:31.637288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:31.637291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:03:31.637311 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:31.637356 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:03:31.637372 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:31.637468 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:31.637599 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 04:03:31.637679 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:03:31.643884 2310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74744bdd2ae04a56bafe275cc204f6d8" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45353 } }
I20260808 04:03:31.644335 2310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:03:31.644428 2310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74744bdd2ae04a56bafe275cc204f6d8, State: Initialized, Role: FOLLOWER
I20260808 04:03:31.645066 2310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74744bdd2ae04a56bafe275cc204f6d8" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45353 } }
I20260808 04:03:31.645303 2310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:03:31.645447 2310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:03:31.645605 2310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:31.647966 2310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74744bdd2ae04a56bafe275cc204f6d8" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45353 } }
I20260808 04:03:31.648521 2310 leader_election.cc:302] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74744bdd2ae04a56bafe275cc204f6d8; no voters:
I20260808 04:03:31.649122 2310 leader_election.cc:288] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:03:31.649317 2333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:31.650034 2333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [term 1 LEADER]: Becoming Leader. State: Replica: 74744bdd2ae04a56bafe275cc204f6d8, State: Running, Role: LEADER
I20260808 04:03:31.650681 2333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74744bdd2ae04a56bafe275cc204f6d8" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45353 } }
I20260808 04:03:31.651806 2310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:31.653903 2336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74744bdd2ae04a56bafe275cc204f6d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74744bdd2ae04a56bafe275cc204f6d8" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45353 } } }
I20260808 04:03:31.653954 2334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74744bdd2ae04a56bafe275cc204f6d8. Latest consensus state: current_term: 1 leader_uuid: "74744bdd2ae04a56bafe275cc204f6d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74744bdd2ae04a56bafe275cc204f6d8" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 45353 } } }
I20260808 04:03:31.654484 2336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:31.654556 2334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:31.656535 2342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:31.661854 2342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:31.671252 2342 catalog_manager.cc:1347] Generated new cluster ID: 1a09432908664177a91d90fa91e7a9ec
I20260808 04:03:31.671396 2342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:31.693373 2342 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:31.693957 2342 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:31.704473 2342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74744bdd2ae04a56bafe275cc204f6d8: Generated new TSK 0
I20260808 04:03:31.705394 2342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:03:31.717407 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2188
W20260808 04:03:31.718608 2301 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40051
W20260808 04:03:31.719369 29679 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2188: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 04:03:31.719563 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 2188
2026-08-08T04:03:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2263 ms)
[----------] 18 tests from AuthTokenIssuingTest (41422 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:31 dist-test-slave-pww0 krb5kdc[2360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:31 dist-test-slave-pww0 krb5kdc[2360](info): set up 2 sockets
Aug 08 04:03:31 dist-test-slave-pww0 krb5kdc[2360](info): commencing operation
krb5kdc: starting...
W20260808 04:03:33.020010 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.003s sys 0.063s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:33 dist-test-slave-pww0 krb5kdc[2360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:03:33.403771 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:33479
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:33479,127.28.251.253:39131,127.28.251.252:43173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:33.780857 2382 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:03:33.781239 2382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:33.781344 2382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:33.781461 2382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:33.788933 2382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:33.789054 2382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:33.789143 2382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:33.789219 2382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:33.789309 2382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:33.799611 2382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:33479,127.28.251.253:39131,127.28.251.252:43173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:33479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:33.800992 2382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:33.802418 2382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:33.814553 2394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:33.814574 2400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:33.814508 2395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:33.815583 2382 server_base.cc:1034] running on GCE node
I20260808 04:03:33.816447 2382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:33.817989 2382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:33.819206 2382 hybrid_clock.cc:648] HybridClock initialized: now 1786161813819129 us; error 55 us; skew 500 ppm
I20260808 04:03:33.819587 2382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:33 dist-test-slave-pww0 krb5kdc[2360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161813, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:33.822930 2382 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:03:33.824517 2382 webserver.cc:460] Webserver started at http://127.28.251.254:43151/ using document root <none> and password file <none>
I20260808 04:03:33.825249 2382 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:33.825345 2382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:33.825639 2382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:33.829167 2382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "86dd613851ce4e4d9a9887432ad37763"
format_stamp: "Formatted at 2026-08-08 04:03:33 on dist-test-slave-pww0"
server_key: "f1e0276b88e637b90f880da48bc65df8"
server_key_iv: "1ee5c8d13d3c1a2d571693dd008fd869"
server_key_version: "encryptionkey@0"
I20260808 04:03:33.829998 2382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "86dd613851ce4e4d9a9887432ad37763"
format_stamp: "Formatted at 2026-08-08 04:03:33 on dist-test-slave-pww0"
server_key: "f1e0276b88e637b90f880da48bc65df8"
server_key_iv: "1ee5c8d13d3c1a2d571693dd008fd869"
server_key_version: "encryptionkey@0"
I20260808 04:03:33.835814 2382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260808 04:03:33.839733 2414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:33.841356 2382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:03:33.841560 2382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "86dd613851ce4e4d9a9887432ad37763"
format_stamp: "Formatted at 2026-08-08 04:03:33 on dist-test-slave-pww0"
server_key: "f1e0276b88e637b90f880da48bc65df8"
server_key_iv: "1ee5c8d13d3c1a2d571693dd008fd869"
server_key_version: "encryptionkey@0"
I20260808 04:03:33.841817 2382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:33.865772 2382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:33.870330 2382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:33.870622 2382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:33.889185 2382 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:33479
I20260808 04:03:33.889215 2499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:33479 every 8 connection(s)
I20260808 04:03:33.890659 2382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:33.892132 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 2382
I20260808 04:03:33.892380 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:33.893229 29679 external_mini_cluster.cc:1428] Setting key dbca0d41a2cc1d9325a2278ea1ec77d2
I20260808 04:03:33.895766 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:33.904994 2501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:03:33 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161813, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:03:33.931212 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:33.908989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:33479 (local address 127.0.0.1:53782)
0808 04:03:33.909887 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:33.909952 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:33.912625 (+ 2673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:33.913648 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:33.913655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:33.914444 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:33.914941 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:33.914973 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:33.916699 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:33.916704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:33.918369 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:33.918385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:33.919519 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:33.919529 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:33.919712 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:33.920404 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:33.920433 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:33.923099 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:33.926925 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:33.926937 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:33.926948 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:33.927403 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:33.927942 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:33.927946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:33.927949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:33.928113 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:33.929138 (+ 1025us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:33.929148 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:33.929392 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 04:03:33.929619 (+ 227us) client_negotiation.cc:240] Negotiation successful
0808 04:03:33.930070 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":393,"threads_started":1}
I20260808 04:03:33.931285 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:33.910420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.28.251.254:33479)
0808 04:03:33.911193 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:33.911201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:33.911255 (+ 54us) server_negotiation.cc:407] Connection header received
0808 04:03:33.913098 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:33.913104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:33.913389 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:33.913566 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:33.915220 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:33.916521 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:33.918543 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:33.919182 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:33.919340 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:33.923371 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:33.923382 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:33.923399 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:33.923442 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:33.926760 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:33.927597 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:33.927608 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:33.927611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:33.927824 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:33.928232 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:33.928237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:33.928239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:33.928955 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:33.929189 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:33.929809 (+ 620us) server_negotiation.cc:299] Negotiation successful
0808 04:03:33.930274 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":544,"thread_start_us":367,"threads_started":1}
WARNING: no policy specified for kudu/127.28.251.253@KRBTEST.COM; defaulting to no policy
I20260808 04:03:33.934674 2501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:03:33.936030 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:33.935522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:37896)
0808 04:03:33.935749 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:33.935884 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260808 04:03:33.936384 2421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.253@KRBTEST.COM" created.
I20260808 04:03:33.938428 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:33.937884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:37912)
0808 04:03:33.938091 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:33.938195 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 04:03:33.938694 2501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:39131: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab.
Entry for principal kudu/127.28.251.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab.
I20260808 04:03:33.979624 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.253:39131
--webserver_interface=127.28.251.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab
--principal=kudu/127.28.251.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:33479,127.28.251.253:39131,127.28.251.252:43173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:03:33.988128 2501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } attempt: 1
I20260808 04:03:33.989552 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:33.988968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:37922)
0808 04:03:33.989219 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:33.989362 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 04:03:33.990770 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:33.990336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:37938)
0808 04:03:33.990503 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:33.990594 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260808 04:03:33.990998 2501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:39131: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
I20260808 04:03:34.059257 2501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } attempt: 2
I20260808 04:03:34.060585 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.060062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:37954)
0808 04:03:34.060337 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.060436 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 04:03:34.061782 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.061369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:37968)
0808 04:03:34.061543 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.061638 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 04:03:34.061991 2501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:39131: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
I20260808 04:03:34.153288 2501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } attempt: 3
I20260808 04:03:34.154735 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.154122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:37982)
0808 04:03:34.154410 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.154542 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 04:03:34.156016 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.155614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:37998)
0808 04:03:34.155771 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.155858 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:03:34.156226 2501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:39131: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
I20260808 04:03:34.299551 2501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } attempt: 4
I20260808 04:03:34.300915 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.300315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:38004)
0808 04:03:34.300653 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.300769 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260808 04:03:34.302119 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.301716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:38016)
0808 04:03:34.301884 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.301984 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 04:03:34.302348 2501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:39131: Network error: Client connection negotiation failed: client connection to 127.28.251.253:39131: connect: Connection refused (error 111)
W20260808 04:03:34.352046 2513 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:03:34.352424 2513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:34.352527 2513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:34.352648 2513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:34.360343 2513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:34.360466 2513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:34.360549 2513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:34.360617 2513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:34.360718 2513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:34.371240 2513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:33479,127.28.251.253:39131,127.28.251.252:43173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab
--principal=kudu/127.28.251.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.253:39131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.251.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:34.372699 2513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:34.374181 2513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:34.386344 2519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:34.386309 2518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:34.386404 2522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:34.387073 2513 server_base.cc:1034] running on GCE node
I20260808 04:03:34.387938 2513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:34.389432 2513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:34.390669 2513 hybrid_clock.cc:648] HybridClock initialized: now 1786161814390591 us; error 45 us; skew 500 ppm
I20260808 04:03:34.391012 2513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:34 dist-test-slave-pww0 krb5kdc[2360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161814, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:34.394456 2513 init.cc:377] Logged in from keytab as kudu/127.28.251.253@KRBTEST.COM (short username kudu)
I20260808 04:03:34.396144 2513 webserver.cc:460] Webserver started at http://127.28.251.253:32821/ using document root <none> and password file <none>
I20260808 04:03:34.396898 2513 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:34.397014 2513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:34.397361 2513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:34.400691 2513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/data/instance:
uuid: "3f0fd9fa5f5046f6812d21ae315c2097"
format_stamp: "Formatted at 2026-08-08 04:03:34 on dist-test-slave-pww0"
server_key: "6d8fed9276dbdd4b8c76b1f53b150d9a"
server_key_iv: "c0e8863e05dce07345e41e9869a69b43"
server_key_version: "encryptionkey@0"
I20260808 04:03:34.401541 2513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/wal/instance:
uuid: "3f0fd9fa5f5046f6812d21ae315c2097"
format_stamp: "Formatted at 2026-08-08 04:03:34 on dist-test-slave-pww0"
server_key: "6d8fed9276dbdd4b8c76b1f53b150d9a"
server_key_iv: "c0e8863e05dce07345e41e9869a69b43"
server_key_version: "encryptionkey@0"
I20260808 04:03:34.407063 2513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:03:34.410797 2531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:34.412387 2513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260808 04:03:34.412568 2513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/wal
uuid: "3f0fd9fa5f5046f6812d21ae315c2097"
format_stamp: "Formatted at 2026-08-08 04:03:34 on dist-test-slave-pww0"
server_key: "6d8fed9276dbdd4b8c76b1f53b150d9a"
server_key_iv: "c0e8863e05dce07345e41e9869a69b43"
server_key_version: "encryptionkey@0"
I20260808 04:03:34.412802 2513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:34.427546 2513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:34.431800 2513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:34.432073 2513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:34.450505 2513 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.253:39131
I20260808 04:03:34.450524 2608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.253:39131 every 8 connection(s)
I20260808 04:03:34.452025 2513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
I20260808 04:03:34.456153 2609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:34.459640 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 2513
I20260808 04:03:34.459852 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-1/wal/instance
I20260808 04:03:34.460462 29679 external_mini_cluster.cc:1428] Setting key 47a5c7b85cf1f761a65c9bdf113f27b0
I20260808 04:03:34.465590 2609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:03:34 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161814, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:03:34.490953 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.469413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.28.251.254:33479)
0808 04:03:34.470033 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:34.470039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:34.471292 (+ 1253us) server_negotiation.cc:407] Connection header received
0808 04:03:34.473884 (+ 2592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:34.473891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:34.473992 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:34.474129 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:34.476304 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.477278 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.479066 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.479789 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.479942 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:34.484511 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:34.484517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:34.484521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:34.484581 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:34.487336 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:34.488416 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:34.488420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:34.488423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:34.488561 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:34.489114 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:34.489118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:34.489120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:34.489412 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:34.489541 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:34.490476 (+ 935us) server_negotiation.cc:299] Negotiation successful
0808 04:03:34.490669 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":311,"threads_started":1}
I20260808 04:03:34.492607 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.470110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:33479 (local address 127.0.0.1:53792)
0808 04:03:34.471061 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.471122 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:34.473458 (+ 2336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:34.474412 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:34.474425 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:34.475165 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:34.476019 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:34.476044 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.477459 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.477464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:34.478855 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:34.478871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.480147 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.480157 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:34.480399 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:34.481161 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:34.481194 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:34.484198 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:34.487571 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:34.487582 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:34.487602 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:34.488280 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:34.488680 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:34.488685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:34.488688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:34.489007 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:34.489556 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:34.489564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:34.490047 (+ 483us) client_negotiation.cc:769] Sending connection context
0808 04:03:34.490433 (+ 386us) client_negotiation.cc:240] Negotiation successful
0808 04:03:34.491041 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":408,"threads_started":1}
I20260808 04:03:34.494740 2609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.251.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.252@KRBTEST.COM" created.
Aug 08 04:03:34 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161814, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
I20260808 04:03:34.511765 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.495651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:38028)
0808 04:03:34.495890 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.495911 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:34.496046 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:34.497283 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:34.497287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:34.497758 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:34.497926 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:34.497936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.500128 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.500133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:34.501050 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:34.501063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.502355 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.502361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:34.502522 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:34.503160 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:34.503179 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:34.505529 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:34.509104 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:34.509109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:34.509113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:34.509504 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:34.509893 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:34.509898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:34.509901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:34.510041 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:34.510683 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:34.510688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:34.510830 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 04:03:34.510953 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 04:03:34.511284 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 04:03:34.511819 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.495886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.28.251.253:39131)
0808 04:03:34.496697 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:34.496706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:34.496740 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:03:34.496803 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:34.496810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:34.497085 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:34.497271 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:34.498118 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.499955 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.501227 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.502185 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.502321 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:34.505761 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:34.505777 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:34.505794 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:34.505828 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:34.508939 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:34.509646 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:34.509655 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:34.509657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:34.509780 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:34.510148 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:34.510152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:34.510154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:34.510548 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:34.510766 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:34.511112 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 04:03:34.511412 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":598,"thread_start_us":433,"threads_started":1}
I20260808 04:03:34.514372 2609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:03:34.515944 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.515216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38338)
0808 04:03:34.515549 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.515750 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
W20260808 04:03:34.516229 2536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
I20260808 04:03:34.517910 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.517539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38342)
0808 04:03:34.517709 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.517796 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:03:34.518157 2609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:43173: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab.
Entry for principal kudu/127.28.251.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab.
I20260808 04:03:34.547147 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.252:43173
--webserver_interface=127.28.251.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab
--principal=kudu/127.28.251.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:33479,127.28.251.253:39131,127.28.251.252:43173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:03:34.567572 2609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } attempt: 1
I20260808 04:03:34.568940 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.568394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38354)
0808 04:03:34.568629 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.568771 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 04:03:34.569998 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.569619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38358)
0808 04:03:34.569788 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.569875 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:03:34.570231 2609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:43173: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
I20260808 04:03:34.601598 2501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } attempt: 5
Aug 08 04:03:34 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161813, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
I20260808 04:03:34.615655 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.602533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.28.251.253:39131)
0808 04:03:34.602720 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:34.602726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:34.602805 (+ 79us) server_negotiation.cc:407] Connection header received
0808 04:03:34.603026 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:34.603030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:34.603102 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:34.603211 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:34.604126 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.604840 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.606055 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.606528 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.606645 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:34.609842 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:34.609848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:34.609851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:34.609887 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:34.612769 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:34.613567 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:34.613572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:34.613575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:34.613718 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:34.614113 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:34.614118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:34.614120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:34.614438 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:34.614581 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:34.615036 (+ 455us) server_negotiation.cc:299] Negotiation successful
0808 04:03:34.615286 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:03:34.615680 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.602424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:38040)
0808 04:03:34.602694 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.602717 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:34.602876 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:34.603262 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:34.603266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:34.603758 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:34.603956 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:34.603967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.604984 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.604990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:34.605891 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:34.605904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:34.606694 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:34.606700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:34.606924 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:34.607578 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:34.607598 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:34.609615 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:34.612980 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:34.612984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:34.612987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:34.613406 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:34.613843 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:34.613849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:34.613852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:34.613999 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:34.614575 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:34.614580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:34.614754 (+ 174us) client_negotiation.cc:769] Sending connection context
0808 04:03:34.614898 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 04:03:34.615285 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 04:03:34.617079 2501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:03:34.618213 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.617771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38374)
0808 04:03:34.617945 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.618055 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 04:03:34.619131 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.618802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38378)
0808 04:03:34.618935 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.619013 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 04:03:34.619351 2501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:43173: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
I20260808 04:03:34.638492 2609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } attempt: 2
I20260808 04:03:34.639775 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.639229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38390)
0808 04:03:34.639450 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.639592 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 04:03:34.641047 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.640666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38398)
0808 04:03:34.640822 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.640906 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 04:03:34.641304 2609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:43173: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
I20260808 04:03:34.670583 2501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } attempt: 1
I20260808 04:03:34.671840 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.671304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38406)
0808 04:03:34.671539 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.671636 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 04:03:34.672873 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.672501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38410)
0808 04:03:34.672654 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.672754 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:03:34.673123 2501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:43173: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
I20260808 04:03:34.732566 2609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } attempt: 3
I20260808 04:03:34.733978 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.733359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38422)
0808 04:03:34.733642 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.733775 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 04:03:34.735123 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.734650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38434)
0808 04:03:34.734923 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.735010 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260808 04:03:34.735342 2609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:43173: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
I20260808 04:03:34.741329 2501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } attempt: 2
I20260808 04:03:34.742443 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.741949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38444)
0808 04:03:34.742184 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.742275 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 04:03:34.743397 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.743045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38460)
0808 04:03:34.743202 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.743283 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 04:03:34.743603 2501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:43173: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
I20260808 04:03:34.849821 2501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } attempt: 3
I20260808 04:03:34.851136 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.850494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38472)
0808 04:03:34.850739 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.850913 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 04:03:34.852344 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.851922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38476)
0808 04:03:34.852101 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.852202 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:03:34.852563 2501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:43173: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
I20260808 04:03:34.878572 2609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } attempt: 4
I20260808 04:03:34.879908 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.879310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38484)
0808 04:03:34.879629 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.879734 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 04:03:34.881031 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:34.880684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38494)
0808 04:03:34.880826 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:34.880906 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:03:34.881246 2609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:43173: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
W20260808 04:03:34.926265 2620 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:03:34.926643 2620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:34.926720 2620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:34.926805 2620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:34.934366 2620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:34.934501 2620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:34.934564 2620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:34.934618 2620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:34.934684 2620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:34.945046 2620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:33479,127.28.251.253:39131,127.28.251.252:43173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab
--principal=kudu/127.28.251.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.252:43173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.251.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:34.946579 2620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:34.948009 2620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:34.960345 2630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:34.960453 2632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:34.960705 2638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:34.961376 2620 server_base.cc:1034] running on GCE node
I20260808 04:03:34.962188 2620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:34.963724 2620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:34.964996 2620 hybrid_clock.cc:648] HybridClock initialized: now 1786161814964911 us; error 42 us; skew 500 ppm
I20260808 04:03:34.965377 2620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:34 dist-test-slave-pww0 krb5kdc[2360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161814, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:34.968715 2620 init.cc:377] Logged in from keytab as kudu/127.28.251.252@KRBTEST.COM (short username kudu)
I20260808 04:03:34.970250 2620 webserver.cc:460] Webserver started at http://127.28.251.252:39991/ using document root <none> and password file <none>
I20260808 04:03:34.971014 2620 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:34.971114 2620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:34.971422 2620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:34.974725 2620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/data/instance:
uuid: "27f661c9414f457f94eeb04b6c7a9fa9"
format_stamp: "Formatted at 2026-08-08 04:03:34 on dist-test-slave-pww0"
server_key: "1331758da56d0bed2b066264bd651dfe"
server_key_iv: "2a2a5039b1f32c84c713b3a736105756"
server_key_version: "encryptionkey@0"
I20260808 04:03:34.975605 2620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/wal/instance:
uuid: "27f661c9414f457f94eeb04b6c7a9fa9"
format_stamp: "Formatted at 2026-08-08 04:03:34 on dist-test-slave-pww0"
server_key: "1331758da56d0bed2b066264bd651dfe"
server_key_iv: "2a2a5039b1f32c84c713b3a736105756"
server_key_version: "encryptionkey@0"
I20260808 04:03:34.981206 2620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:03:34.985145 2649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:34.986773 2620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:03:34.986945 2620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/wal
uuid: "27f661c9414f457f94eeb04b6c7a9fa9"
format_stamp: "Formatted at 2026-08-08 04:03:34 on dist-test-slave-pww0"
server_key: "1331758da56d0bed2b066264bd651dfe"
server_key_iv: "2a2a5039b1f32c84c713b3a736105756"
server_key_version: "encryptionkey@0"
I20260808 04:03:34.987152 2620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:35.011780 2620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:35.016841 2620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:35.017107 2620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:35.029774 2501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } attempt: 4
I20260808 04:03:35.031028 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.030500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38504)
0808 04:03:35.030756 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:35.030864 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 04:03:35.032399 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.031905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38516)
0808 04:03:35.032106 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:35.032206 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260808 04:03:35.032647 2501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:43173: Network error: Client connection negotiation failed: client connection to 127.28.251.252:43173: connect: Connection refused (error 111)
I20260808 04:03:35.038952 2620 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.252:43173
I20260808 04:03:35.038975 2741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.252:43173 every 8 connection(s)
I20260808 04:03:35.040482 2620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
I20260808 04:03:35.044574 2742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:35.047353 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 2620
I20260808 04:03:35.047562 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786161430375636-29679-0/minicluster-data/master-2/wal/instance
I20260808 04:03:35.048028 29679 external_mini_cluster.cc:1428] Setting key 391b5fa78f4721c7012c484e974f37d4
I20260808 04:03:35.053119 2742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:03:35 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
Aug 08 04:03:35 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161814, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:03:35.073081 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.056290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.28.251.254:33479)
0808 04:03:35.056918 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.056924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.057440 (+ 516us) server_negotiation.cc:407] Connection header received
0808 04:03:35.057659 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.057663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.057751 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.057849 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.059087 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.060047 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.061857 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.062517 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.062673 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.066411 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:35.066416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:35.066420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:35.066468 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:35.069782 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.070652 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.070657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.070660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.070815 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.071191 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.071196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.071198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.071530 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:35.071674 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:35.072403 (+ 729us) server_negotiation.cc:299] Negotiation successful
0808 04:03:35.072639 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":316,"threads_started":1}
I20260808 04:03:35.073515 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.056287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:33479 (local address 127.0.0.1:53800)
0808 04:03:35.057286 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:35.057332 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:35.057503 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:35.057922 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:35.057928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:35.058644 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:35.058910 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:35.058919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.060210 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.060216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:35.061672 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:35.061686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.062771 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.062785 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:35.062919 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.063566 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:35.063590 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:35.066169 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:35.069999 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:35.070012 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:35.070021 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:35.070516 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:35.070937 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:35.070942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:35.070945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:35.071080 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:35.071666 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:35.071674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:35.071900 (+ 226us) client_negotiation.cc:769] Sending connection context
0808 04:03:35.072082 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 04:03:35.072466 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":37,"thread_start_us":564,"threads_started":1}
I20260808 04:03:35.073544 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.050234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.28.251.252:43173)
0808 04:03:35.051195 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.051210 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.051268 (+ 58us) server_negotiation.cc:407] Connection header received
0808 04:03:35.053269 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.053296 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.053807 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.054230 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.056032 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.058009 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.059624 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.060449 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.060655 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.065475 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:35.065488 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:35.065507 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:35.065544 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:35.068651 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.069662 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.069671 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.069679 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.069878 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.070355 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.070360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.070362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.070968 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:35.071306 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:35.071831 (+ 525us) server_negotiation.cc:299] Negotiation successful
0808 04:03:35.072323 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":406,"threads_started":1}
I20260808 04:03:35.075300 2742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:03:35 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161814, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
Aug 08 04:03:35 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:03:35.090821 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.076211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:39131 (local address 127.0.0.1:38044)
0808 04:03:35.076391 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:35.076411 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:35.076523 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:35.076872 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:35.076877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:35.077357 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:35.077566 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:35.077576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.078623 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.078629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:35.079701 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:35.079718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.080464 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.080471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:35.080704 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.081426 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:35.081446 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:35.084146 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:35.087953 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:35.087958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:35.087960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:35.088399 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:35.088900 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:35.088905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:35.088907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:35.089072 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:35.089750 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:35.089754 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:35.089933 (+ 179us) client_negotiation.cc:769] Sending connection context
0808 04:03:35.090107 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 04:03:35.090492 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 04:03:35.090827 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.076320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.28.251.253:39131)
0808 04:03:35.076508 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.076514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.076534 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:03:35.076631 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.076637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.076707 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.076816 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.077735 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.078452 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.079853 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.080298 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.080397 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.084421 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:35.084428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:35.084431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:35.084471 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:35.087795 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.088536 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.088543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.088546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.088739 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.089191 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.089196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.089199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.089600 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:35.089771 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:35.090208 (+ 437us) server_negotiation.cc:299] Negotiation successful
0808 04:03:35.090418 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:03:35.092067 2742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:03:35.093662 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.077170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.28.251.254:33479)
0808 04:03:35.077326 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.077331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.077347 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:03:35.077466 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.077470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.077537 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.077613 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.078936 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.079651 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.081165 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.081650 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.081751 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.086442 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:35.086450 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:35.086455 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:35.086504 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:35.089795 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.090707 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.090713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.090716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.090879 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.091412 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.091418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.091421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.091824 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:35.092081 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:35.093045 (+ 964us) server_negotiation.cc:299] Negotiation successful
0808 04:03:35.093320 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 04:03:35 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161814, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
Aug 08 04:03:35 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
I20260808 04:03:35.106833 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.092817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38528)
0808 04:03:35.092986 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:35.093004 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:35.093105 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:35.093459 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:35.093462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:35.093906 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:35.094072 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:35.094082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.095024 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.095029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:35.096253 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:35.096271 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.097070 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.097076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:35.097312 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.097936 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:35.097953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:35.100490 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:35.104201 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:35.104205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:35.104208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:35.104754 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:35.105190 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:35.105194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:35.105198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:35.105334 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:35.105839 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:35.105843 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:35.105984 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 04:03:35.106102 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 04:03:35.106404 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 04:03:35.106856 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.093024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.28.251.252:43173)
0808 04:03:35.093199 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.093205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.093223 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:03:35.093274 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.093278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.093350 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.093457 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.094202 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.094903 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.096457 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.096899 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.097014 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.100755 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:35.100762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:35.100767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:35.100807 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:35.104014 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.104893 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.104897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.104900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.105080 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.105438 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.105443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.105447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.105726 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:35.105858 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:35.106215 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 04:03:35.106570 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":57}
I20260808 04:03:35.111603 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.095331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38058 (local address 127.28.251.253:39131)
0808 04:03:35.095516 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.095522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.095543 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:03:35.095738 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.095741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.095802 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.095876 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.097249 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.098141 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.099746 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.100455 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.100599 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.105917 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:35.105924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:35.105928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:35.105965 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:35.108754 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.109496 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.109501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.109504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.109692 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.110095 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.110099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.110101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.110401 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:35.110675 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:35.110981 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 04:03:35.111243 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:03:35.112178 2742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9: Bootstrap starting.
I20260808 04:03:35.116078 2742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:35.117276 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:35.117550 2742 log.cc:824] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:35.120714 2742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9: No bootstrap required, opened a new log
I20260808 04:03:35.120473 2462 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "86dd613851ce4e4d9a9887432ad37763"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53804
I20260808 04:03:35.130501 2742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } }
I20260808 04:03:35.131026 2742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f661c9414f457f94eeb04b6c7a9fa9, State: Initialized, Role: FOLLOWER
I20260808 04:03:35.131676 2742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } }
I20260808 04:03:35.132898 2752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } } }
I20260808 04:03:35.133353 2752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:35.134377 2742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:03:35.144100 2771 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:03:35.144260 2771 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:03:35.142014 2620 master_runner.cc:186] Error getting master registration for 127.28.251.254:33479: OK, instance_id { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" instance_seqno: 1786161813883874 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:03:35.153926 2620 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:03:35.180526 2609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } attempt: 5
Aug 08 04:03:35 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161814, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
I20260808 04:03:35.197788 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.181353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38536)
0808 04:03:35.181592 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:35.181615 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:35.181768 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:35.182345 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:35.182350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:35.182880 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:35.183096 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:35.183108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.184465 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.184472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:35.185395 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:35.185410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.186539 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.186546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:35.186784 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.187428 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:35.187446 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:35.190165 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:35.194517 (+ 4352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:35.194524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:35.194528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:35.195131 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:35.195586 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:35.195591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:35.195595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:35.195840 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:35.196633 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:35.196639 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:35.196835 (+ 196us) client_negotiation.cc:769] Sending connection context
0808 04:03:35.197003 (+ 168us) client_negotiation.cc:240] Negotiation successful
0808 04:03:35.197356 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 04:03:35.197846 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.181478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.28.251.252:43173)
0808 04:03:35.181744 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.181750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.181777 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:03:35.181958 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.181964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.182052 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.182152 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.183273 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.184289 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.185562 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.186372 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.186503 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.190425 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:35.190432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:35.190438 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:35.190488 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:35.194320 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.195270 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.195275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.195279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.195447 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.195956 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.195960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.195964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.196483 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:35.196635 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:35.197173 (+ 538us) server_negotiation.cc:299] Negotiation successful
0808 04:03:35.197447 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 04:03:35.204090 2609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097: Bootstrap starting.
I20260808 04:03:35.208400 2609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:35.209817 2609 log.cc:824] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:35.213615 2609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097: No bootstrap required, opened a new log
I20260808 04:03:35.225276 2609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } }
I20260808 04:03:35.225752 2609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f0fd9fa5f5046f6812d21ae315c2097, State: Initialized, Role: FOLLOWER
I20260808 04:03:35.226341 2609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } }
I20260808 04:03:35.227818 2778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } } }
I20260808 04:03:35.228579 2778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:35.229112 2609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:03:35.254611 2797 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:03:35.254810 2797 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:03:35.240664 2513 master_runner.cc:186] Error getting master registration for 127.28.251.254:33479: OK, instance_id { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" instance_seqno: 1786161813883874 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:03:35.257041 2513 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 04:03:35.269668 2661 tablet.cc:2369] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:03:35.309878 2501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } attempt: 5
Aug 08 04:03:35 dist-test-slave-pww0 krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161813, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
I20260808 04:03:35.330238 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.310723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:43173 (local address 127.0.0.1:38544)
0808 04:03:35.311042 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:35.311061 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:35.311212 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:35.311695 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:35.311701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:35.312262 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:35.312502 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:35.312515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.314074 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.314080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:35.315660 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:35.315680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.316808 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.316815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:35.317116 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.317885 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:35.317988 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:35.321031 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:35.325820 (+ 4789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:35.325828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:35.325832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:35.326467 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:35.327017 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:35.327023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:35.327027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:35.327247 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:35.328059 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:35.328064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:35.328270 (+ 206us) client_negotiation.cc:769] Sending connection context
0808 04:03:35.328424 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 04:03:35.328758 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260808 04:03:35.330502 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.310871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.28.251.252:43173)
0808 04:03:35.311070 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.311076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.311129 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:03:35.311359 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.311365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.311512 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.311654 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.312703 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.313913 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.315867 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.316640 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.316777 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.321347 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:35.321355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:35.321360 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:35.321408 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:35.325571 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.326656 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.326663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.326667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.326862 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.327395 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.327400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.327404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.327830 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:35.327986 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:35.329299 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0808 04:03:35.329565 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:03:35.337659 2501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763: Bootstrap starting.
I20260808 04:03:35.342235 2501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:35.343784 2501 log.cc:824] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:35.347995 2501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763: No bootstrap required, opened a new log
I20260808 04:03:35.356482 2501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } }
I20260808 04:03:35.356949 2501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86dd613851ce4e4d9a9887432ad37763, State: Initialized, Role: FOLLOWER
I20260808 04:03:35.357543 2501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } }
I20260808 04:03:35.358994 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } } }
I20260808 04:03:35.359769 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:35.360348 2501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:35.360436 2462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 04:03:35.360666 2462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:35.364195 2462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } }
I20260808 04:03:35.367000 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86dd613851ce4e4d9a9887432ad37763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3f0fd9fa5f5046f6812d21ae315c2097"
I20260808 04:03:35.367597 2568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:35.371371 2568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86dd613851ce4e4d9a9887432ad37763 in term 1.
I20260808 04:03:35.372463 2421 leader_election.cc:302] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f0fd9fa5f5046f6812d21ae315c2097, 86dd613851ce4e4d9a9887432ad37763; no voters:
W20260808 04:03:35.373960 2425 tablet.cc:2369] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:03:35.374408 2699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86dd613851ce4e4d9a9887432ad37763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f661c9414f457f94eeb04b6c7a9fa9"
I20260808 04:03:35.375109 2699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:35.379050 2699 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86dd613851ce4e4d9a9887432ad37763 in term 1.
I20260808 04:03:35.380010 2462 leader_election.cc:288] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [CANDIDATE]: Term 1 election: Requested vote from peers 3f0fd9fa5f5046f6812d21ae315c2097 (127.28.251.253:39131), 27f661c9414f457f94eeb04b6c7a9fa9 (127.28.251.252:43173)
I20260808 04:03:35.387596 2803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:35.397380 2803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [term 1 LEADER]: Becoming Leader. State: Replica: 86dd613851ce4e4d9a9887432ad37763, State: Running, Role: LEADER
I20260808 04:03:35.398418 2803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } }
I20260808 04:03:35.409569 2819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86dd613851ce4e4d9a9887432ad37763. Latest consensus state: current_term: 1 leader_uuid: "86dd613851ce4e4d9a9887432ad37763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } } }
I20260808 04:03:35.410209 2819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:35.415266 2838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260808 04:03:35.434962 2539 tablet.cc:2369] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:03:35.437777 2838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:03:35.450407 2850 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:03:35.469589 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.447287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.28.251.252:43173)
0808 04:03:35.447551 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.447556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.451224 (+ 3668us) server_negotiation.cc:407] Connection header received
0808 04:03:35.451534 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.451541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.451632 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.451711 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.453725 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.454842 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.456918 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.457640 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.457750 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.461959 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:35.461966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:35.461972 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:35.462021 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:35.465853 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.466887 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.466894 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.466897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.467069 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.467600 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.467605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.467609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.467989 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:35.468303 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:35.468927 (+ 624us) server_negotiation.cc:299] Negotiation successful
0808 04:03:35.469209 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:03:35.477843 2699 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [term 1 FOLLOWER]: Refusing update from remote peer 86dd613851ce4e4d9a9887432ad37763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:03:35.479174 2819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:03:35.488624 2568 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [term 1 FOLLOWER]: Refusing update from remote peer 86dd613851ce4e4d9a9887432ad37763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:03:35.489877 2819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:03:35.503221 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.430062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.28.251.254:33479)
0808 04:03:35.430281 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.430287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.447441 (+ 17154us) server_negotiation.cc:407] Connection header received
0808 04:03:35.459595 (+ 12154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.459601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.459709 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.459789 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.472159 (+ 12370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.473277 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.478625 (+ 5348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.479308 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.479416 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.488049 (+ 8633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:35.488057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:35.488061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:35.488116 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:35.491757 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.498247 (+ 6490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.498256 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.498260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.498492 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.499824 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.499830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.499834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.500274 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:35.500585 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:35.502485 (+ 1900us) server_negotiation.cc:299] Negotiation successful
0808 04:03:35.502797 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:03:35.504979 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:35.429674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.28.251.253:39131)
0808 04:03:35.429876 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.429882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.446895 (+ 17013us) server_negotiation.cc:407] Connection header received
0808 04:03:35.447606 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.447613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.447720 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.447804 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.475645 (+ 27841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.476761 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.480606 (+ 3845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:35.481329 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:35.481446 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:35.492876 (+ 11430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:35.492884 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:35.492889 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:35.492947 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:35.497097 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.499053 (+ 1956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.499061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.499064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.499252 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:35.501155 (+ 1903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:35.501161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:35.501165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:35.501591 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:35.501860 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:35.504306 (+ 2446us) server_negotiation.cc:299] Negotiation successful
0808 04:03:35.504595 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:03:35.522290 2752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86dd613851ce4e4d9a9887432ad37763. Latest consensus state: current_term: 1 leader_uuid: "86dd613851ce4e4d9a9887432ad37763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } } }
I20260808 04:03:35.522675 2752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:35.524499 2852 mvcc.cc:205] Tried to move back new op lower bound from 7316118796183027712 to 7316118795890917376. Current Snapshot: MvccSnapshot[applied={T|T < 7316118796183027712}]
I20260808 04:03:35.526268 2778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86dd613851ce4e4d9a9887432ad37763. Latest consensus state: current_term: 1 leader_uuid: "86dd613851ce4e4d9a9887432ad37763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } } }
I20260808 04:03:35.526639 2778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:35.532092 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86dd613851ce4e4d9a9887432ad37763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } } }
I20260808 04:03:35.532490 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:35.532850 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86dd613851ce4e4d9a9887432ad37763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } } }
I20260808 04:03:35.533103 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:35.534650 2838 catalog_manager.cc:1347] Generated new cluster ID: 2e48cb70652a48248ab3498e7d840302
I20260808 04:03:35.534762 2838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:35.536252 2778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86dd613851ce4e4d9a9887432ad37763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } } }
I20260808 04:03:35.536526 2778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:35.537056 2858 mvcc.cc:205] Tried to move back new op lower bound from 7316118796183027712 to 7316118795890917376. Current Snapshot: MvccSnapshot[applied={T|T < 7316118796183027712}]
I20260808 04:03:35.537839 2752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86dd613851ce4e4d9a9887432ad37763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd613851ce4e4d9a9887432ad37763" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 33479 } } peers { permanent_uuid: "3f0fd9fa5f5046f6812d21ae315c2097" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 39131 } } peers { permanent_uuid: "27f661c9414f457f94eeb04b6c7a9fa9" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 43173 } } }
I20260808 04:03:35.538103 2752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:35.562297 2838 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:35.564278 2838 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:35.578097 2838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86dd613851ce4e4d9a9887432ad37763: Generated new TSK 0
I20260808 04:03:35.579447 2838 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:03:35.704289 29679 mini_kdc.cc:348] Time spent kdestroy: real 0.108s user 0.001s sys 0.036s
W20260808 04:03:35.723531 2749 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:35.709256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.28.251.254:33479)
0808 04:03:35.709556 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.709563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.721242 (+ 11679us) server_negotiation.cc:407] Connection header received
0808 04:03:35.721761 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.721768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.721856 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.721927 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.723216 (+ 1289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53822: BlockingRecv error: recv got EOF from 127.0.0.1:53822 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260808 04:03:35.723616 2616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:35.709257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.28.251.253:39131)
0808 04:03:35.709526 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.709534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.720605 (+ 11071us) server_negotiation.cc:407] Connection header received
0808 04:03:35.721131 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.721138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.721242 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.721353 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.723323 (+ 1970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38072: BlockingRecv error: recv got EOF from 127.0.0.1:38072 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260808 04:03:35.726310 2747 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:35.720815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.28.251.252:43173)
0808 04:03:35.721017 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.721023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.724535 (+ 3512us) server_negotiation.cc:407] Connection header received
0808 04:03:35.724805 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.724811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.724905 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.724998 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.726047 (+ 1049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38552: BlockingRecv error: recv got EOF from 127.0.0.1:38552 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260808 04:03:35.736079 2616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:35.732871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.28.251.253:39131)
0808 04:03:35.733092 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.733098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.734124 (+ 1026us) server_negotiation.cc:407] Connection header received
0808 04:03:35.734402 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.734409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.734614 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.734771 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.735729 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38076: BlockingRecv error: recv got EOF from 127.0.0.1:38076 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260808 04:03:35.744566 2616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:35.741932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.28.251.253:39131)
0808 04:03:35.742127 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.742133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.742687 (+ 554us) server_negotiation.cc:407] Connection header received
0808 04:03:35.742999 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.743005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.743102 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.743180 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.744308 (+ 1128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38090: BlockingRecv error: recv got EOF from 127.0.0.1:38090 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 04:03:35.753837 2616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:35.751419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.28.251.253:39131)
0808 04:03:35.751646 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.751652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.752228 (+ 576us) server_negotiation.cc:407] Connection header received
0808 04:03:35.752432 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.752438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.752576 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.752698 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.753565 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38098: BlockingRecv error: recv got EOF from 127.0.0.1:38098 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260808 04:03:35.776676 2616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:35.773992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.28.251.253:39131)
0808 04:03:35.774238 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.774245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.774877 (+ 632us) server_negotiation.cc:407] Connection header received
0808 04:03:35.775105 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.775122 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.775229 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.775339 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.776402 (+ 1063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38108: BlockingRecv error: recv got EOF from 127.0.0.1:38108 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260808 04:03:35.791632 2616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:35.788983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.28.251.253:39131)
0808 04:03:35.789183 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.789189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.789805 (+ 616us) server_negotiation.cc:407] Connection header received
0808 04:03:35.790060 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.790066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.790192 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.790300 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.791309 (+ 1009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38120: BlockingRecv error: recv got EOF from 127.0.0.1:38120 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 04:03:35.817422 2616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:35.815164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.28.251.253:39131)
0808 04:03:35.815433 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.815440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.815898 (+ 458us) server_negotiation.cc:407] Connection header received
0808 04:03:35.816126 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.816131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.816241 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.816351 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.817162 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38132: BlockingRecv error: recv got EOF from 127.0.0.1:38132 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260808 04:03:35.856015 2616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:35.853494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.28.251.253:39131)
0808 04:03:35.853769 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.853775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.854246 (+ 471us) server_negotiation.cc:407] Connection header received
0808 04:03:35.854528 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.854533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.854631 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.854733 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.855724 (+ 991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38136: BlockingRecv error: recv got EOF from 127.0.0.1:38136 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260808 04:03:35.927526 2616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:35.925121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.28.251.253:39131)
0808 04:03:35.925415 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:35.925421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:35.925787 (+ 366us) server_negotiation.cc:407] Connection header received
0808 04:03:35.926035 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:35.926039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:35.926119 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:35.926213 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:35.927269 (+ 1056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38144: BlockingRecv error: recv got EOF from 127.0.0.1:38144 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260808 04:03:36.061302 2616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:36.059180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.28.251.253:39131)
0808 04:03:36.059408 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:36.059415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:36.059796 (+ 381us) server_negotiation.cc:407] Connection header received
0808 04:03:36.060025 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:36.060030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:36.060108 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:36.060197 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:36.061061 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38158: BlockingRecv error: recv got EOF from 127.0.0.1:38158 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:03:36.146530 2771 catalog_manager.cc:1259] Loaded cluster ID: 2e48cb70652a48248ab3498e7d840302
I20260808 04:03:36.146644 2771 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9: loading cluster ID for follower catalog manager: success
I20260808 04:03:36.149475 2771 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9: acquiring CA information for follower catalog manager: success
I20260808 04:03:36.151684 2771 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27f661c9414f457f94eeb04b6c7a9fa9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:03:36.257138 2797 catalog_manager.cc:1259] Loaded cluster ID: 2e48cb70652a48248ab3498e7d840302
I20260808 04:03:36.257253 2797 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097: loading cluster ID for follower catalog manager: success
I20260808 04:03:36.260245 2797 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097: acquiring CA information for follower catalog manager: success
I20260808 04:03:36.262542 2797 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f0fd9fa5f5046f6812d21ae315c2097: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:03:36.327412 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:36.321429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.28.251.253:39131)
0808 04:03:36.321717 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:36.321722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:36.321853 (+ 131us) server_negotiation.cc:407] Connection header received
0808 04:03:36.322046 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:36.322051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:36.322222 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:36.322355 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:36.322927 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:36.323705 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:36.325114 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:36.325533 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:36.325625 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:36.326175 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:36.326759 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:36.326885 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:36.327002 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 04:03:36.327083 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260808 04:03:36.340838 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:36.334483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38566 (local address 127.28.251.252:43173)
0808 04:03:36.334985 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:36.334989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:36.335012 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:03:36.335242 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:36.335246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:36.335519 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:36.335690 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:36.336124 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:36.336921 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:36.338450 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:36.338853 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:36.338954 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:36.339517 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:36.340163 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:36.340288 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:36.340414 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 04:03:36.340493 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":251,"threads_started":1}
I20260808 04:03:36.355207 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:36.347720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.28.251.253:39131)
0808 04:03:36.347942 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:36.347949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:36.348993 (+ 1044us) server_negotiation.cc:407] Connection header received
0808 04:03:36.349292 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:36.349298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:36.349388 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:36.349451 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:36.350297 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:36.351239 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:36.353028 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:36.353668 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:36.353785 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:36.354351 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:36.354483 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:36.354577 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:36.354804 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 04:03:36.354898 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 04:03:36.355666 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:36.347665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.28.251.254:33479)
0808 04:03:36.348292 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:36.348298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:36.348319 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:03:36.348460 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:36.348465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:36.348716 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:36.348940 (+ 224us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:36.349804 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:36.350733 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:36.352930 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:36.353535 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:36.353670 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:36.354197 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:36.354710 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:36.354824 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:36.355235 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 04:03:36.355331 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":395,"threads_started":1}
I20260808 04:03:36.356113 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:36.349266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.28.251.252:43173)
0808 04:03:36.349458 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:36.349464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:36.350100 (+ 636us) server_negotiation.cc:407] Connection header received
0808 04:03:36.350268 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:36.350274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:36.350372 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:36.350465 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:36.351073 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:36.352203 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:36.353911 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:36.354466 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:36.354584 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:36.355360 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:36.355523 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:36.355620 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:36.355808 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 04:03:36.355897 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:03:36.361771 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2382
I20260808 04:03:36.362807 2493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:36.475003 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 2382
W20260808 04:03:36.503525 2535 connection.cc:535] server connection from 127.0.0.1:38040 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:03:36.503630 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2513
I20260808 04:03:36.504739 2602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:36.616648 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 2513
I20260808 04:03:36.643788 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 2620
I20260808 04:03:36.644991 2735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:36.766040 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 2620
2026-08-08T04:03:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5052 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:03:36 dist-test-slave-pww0 krb5kdc[2992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:03:36 dist-test-slave-pww0 krb5kdc[2992](info): set up 2 sockets
Aug 08 04:03:36 dist-test-slave-pww0 krb5kdc[2992](info): commencing operation
krb5kdc: starting...
W20260808 04:03:38.089916 29679 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.001s sys 0.067s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/test-user.keytab.
Aug 08 04:03:38 dist-test-slave-pww0 krb5kdc[2992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:03:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:03:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
Entry for principal kudu/127.28.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab.
I20260808 04:03:38.474831 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43523
--webserver_interface=127.28.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:43523,127.28.251.253:41283,127.28.251.252:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:03:38.843583 3017 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:03:38.843932 3017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:38.844008 3017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:38.844092 3017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:38.851923 3017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:38.852032 3017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:38.852092 3017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:38.852144 3017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:38.852206 3017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:38.862563 3017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:43523,127.28.251.253:41283,127.28.251.252:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.254.keytab
--principal=kudu/127.28.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.254:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:38.863951 3017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:38.865319 3017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:38.877557 3023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:38.877609 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:38.877496 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:38.878525 3017 server_base.cc:1034] running on GCE node
I20260808 04:03:38.879355 3017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:38.880916 3017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:38.882174 3017 hybrid_clock.cc:648] HybridClock initialized: now 1786161818882089 us; error 43 us; skew 500 ppm
I20260808 04:03:38.882529 3017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:38 dist-test-slave-pww0 krb5kdc[2992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161818, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:38.889608 3017 init.cc:377] Logged in from keytab as kudu/127.28.251.254@KRBTEST.COM (short username kudu)
I20260808 04:03:38.891391 3017 webserver.cc:460] Webserver started at http://127.28.251.254:36787/ using document root <none> and password file <none>
I20260808 04:03:38.892233 3017 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:38.892354 3017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:38.892688 3017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:38.895967 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/data/instance:
uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747"
format_stamp: "Formatted at 2026-08-08 04:03:38 on dist-test-slave-pww0"
server_key: "b2e6f9ef93f397033c9def857be1936f"
server_key_iv: "af5a25fa2b5df458bd72cc86b59a93db"
server_key_version: "encryptionkey@0"
I20260808 04:03:38.896818 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/wal/instance:
uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747"
format_stamp: "Formatted at 2026-08-08 04:03:38 on dist-test-slave-pww0"
server_key: "b2e6f9ef93f397033c9def857be1936f"
server_key_iv: "af5a25fa2b5df458bd72cc86b59a93db"
server_key_version: "encryptionkey@0"
I20260808 04:03:38.902501 3017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:03:38.906333 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:38.907796 3017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:03:38.907975 3017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/wal
uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747"
format_stamp: "Formatted at 2026-08-08 04:03:38 on dist-test-slave-pww0"
server_key: "b2e6f9ef93f397033c9def857be1936f"
server_key_iv: "af5a25fa2b5df458bd72cc86b59a93db"
server_key_version: "encryptionkey@0"
I20260808 04:03:38.908198 3017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:38.927898 3017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:38.932722 3017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:38.933035 3017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:38.951762 3017 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.254:43523
I20260808 04:03:38.951781 3113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.254:43523 every 8 connection(s)
I20260808 04:03:38.953301 3017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/data/info.pb
I20260808 04:03:38.953648 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 3017
I20260808 04:03:38.953838 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-0/wal/instance
I20260808 04:03:38.954334 29679 external_mini_cluster.cc:1428] Setting key 98ccd3c5b9d9bd2916b7c5af51cbb945
I20260808 04:03:38.957522 3114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:38.964603 3114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:03:38 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161818, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:03:38.989248 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:38.969620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.28.251.254:43523)
0808 04:03:38.970340 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:38.970362 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:38.970432 (+ 70us) server_negotiation.cc:407] Connection header received
0808 04:03:38.971761 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:38.971766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:38.972034 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:38.972244 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:38.973603 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:38.974825 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:38.976320 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:38.976894 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:38.977079 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:38.981099 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:38.981109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:38.981125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:38.981161 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:38.984262 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:38.985193 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:38.985204 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:38.985207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:38.985455 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:38.985990 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:38.985996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:38.985999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:38.986729 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:38.986912 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:38.987720 (+ 808us) server_negotiation.cc:299] Negotiation successful
0808 04:03:38.988056 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":508,"thread_start_us":330,"threads_started":1}
I20260808 04:03:38.989310 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:38.968193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43523 (local address 127.0.0.1:46084)
0808 04:03:38.969070 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:38.969120 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:38.971312 (+ 2192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:38.972265 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:38.972273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:38.973022 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:38.973380 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:38.973400 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:38.974978 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:38.974982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:38.976177 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:38.976192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:38.977129 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:38.977136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:38.977421 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:38.978185 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:38.978218 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:38.980818 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:38.984429 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:38.984444 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:38.984458 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:38.984958 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:38.985617 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:38.985622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:38.985624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:38.985839 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:38.986915 (+ 1076us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:38.986925 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:38.987193 (+ 268us) client_negotiation.cc:769] Sending connection context
0808 04:03:38.987570 (+ 377us) client_negotiation.cc:240] Negotiation successful
0808 04:03:38.988254 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":167,"thread_start_us":375,"threads_started":1}
WARNING: no policy specified for kudu/127.28.251.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.253@KRBTEST.COM" created.
I20260808 04:03:38.992919 3114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:03:38.994237 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:38.993779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40226)
0808 04:03:38.993993 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:38.994113 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260808 04:03:38.994553 3042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
I20260808 04:03:38.996335 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:38.995957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40232)
0808 04:03:38.996110 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:38.996182 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 04:03:38.996539 3114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:41283: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab.
Entry for principal kudu/127.28.251.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab.
I20260808 04:03:39.035295 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.253:41283
--webserver_interface=127.28.251.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab
--principal=kudu/127.28.251.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:43523,127.28.251.253:41283,127.28.251.252:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:03:39.045917 3114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } attempt: 1
I20260808 04:03:39.047163 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.046671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40240)
0808 04:03:39.046881 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.047021 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 04:03:39.049094 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.048513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40254)
0808 04:03:39.048798 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.048898 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260808 04:03:39.049436 3114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:41283: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
I20260808 04:03:39.117780 3114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } attempt: 2
I20260808 04:03:39.118988 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.118471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40256)
0808 04:03:39.118730 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.118836 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 04:03:39.120389 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.119933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40264)
0808 04:03:39.120121 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.120218 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 04:03:39.120635 3114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:41283: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
I20260808 04:03:39.211889 3114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } attempt: 3
I20260808 04:03:39.213155 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.212656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40266)
0808 04:03:39.212865 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.212992 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 04:03:39.214278 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.213877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40280)
0808 04:03:39.214038 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.214116 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:03:39.214468 3114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:41283: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
I20260808 04:03:39.357752 3114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } attempt: 4
I20260808 04:03:39.358995 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.358414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40286)
0808 04:03:39.358755 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.358851 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260808 04:03:39.360137 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.359756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40296)
0808 04:03:39.359910 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.360000 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 04:03:39.360383 3114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.253:41283: Network error: Client connection negotiation failed: client connection to 127.28.251.253:41283: connect: Connection refused (error 111)
W20260808 04:03:39.405334 3122 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:03:39.405727 3122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:39.405833 3122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:39.405951 3122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:39.413554 3122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:39.413710 3122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:39.413825 3122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:39.413893 3122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:39.413986 3122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:39.424455 3122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:43523,127.28.251.253:41283,127.28.251.252:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.253.keytab
--principal=kudu/127.28.251.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.253:41283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.251.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:39.425925 3122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:39.427318 3122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:39.440078 3132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:39.440062 3136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:39.440099 3130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:39.449380 3122 server_base.cc:1034] running on GCE node
I20260808 04:03:39.450363 3122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:39.452095 3122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:39.453428 3122 hybrid_clock.cc:648] HybridClock initialized: now 1786161819453335 us; error 51 us; skew 500 ppm
I20260808 04:03:39.453872 3122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:39 dist-test-slave-pww0 krb5kdc[2992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161819, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:39.457476 3122 init.cc:377] Logged in from keytab as kudu/127.28.251.253@KRBTEST.COM (short username kudu)
I20260808 04:03:39.459347 3122 webserver.cc:460] Webserver started at http://127.28.251.253:43909/ using document root <none> and password file <none>
I20260808 04:03:39.460201 3122 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:39.460319 3122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:39.460675 3122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:39.464064 3122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/data/instance:
uuid: "22b136c297e94274a4177478ff74e202"
format_stamp: "Formatted at 2026-08-08 04:03:39 on dist-test-slave-pww0"
server_key: "df6241a386be1b2fe17148945473459b"
server_key_iv: "d60b1976ecf46606af6c36d9f3575296"
server_key_version: "encryptionkey@0"
I20260808 04:03:39.464951 3122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/wal/instance:
uuid: "22b136c297e94274a4177478ff74e202"
format_stamp: "Formatted at 2026-08-08 04:03:39 on dist-test-slave-pww0"
server_key: "df6241a386be1b2fe17148945473459b"
server_key_iv: "d60b1976ecf46606af6c36d9f3575296"
server_key_version: "encryptionkey@0"
I20260808 04:03:39.470719 3122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:03:39.474566 3151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:39.476090 3122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:03:39.476274 3122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/wal
uuid: "22b136c297e94274a4177478ff74e202"
format_stamp: "Formatted at 2026-08-08 04:03:39 on dist-test-slave-pww0"
server_key: "df6241a386be1b2fe17148945473459b"
server_key_iv: "d60b1976ecf46606af6c36d9f3575296"
server_key_version: "encryptionkey@0"
I20260808 04:03:39.476504 3122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:39.506662 3122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:39.511346 3122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:39.511662 3122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:39.530993 3122 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.253:41283
I20260808 04:03:39.531015 3226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.253:41283 every 8 connection(s)
I20260808 04:03:39.532437 3122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/data/info.pb
I20260808 04:03:39.535794 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 3122
I20260808 04:03:39.536000 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-1/wal/instance
I20260808 04:03:39.536583 29679 external_mini_cluster.cc:1428] Setting key f5486b89ac943105cb5b62be7e596fb1
I20260808 04:03:39.536864 3227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:39.546260 3227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:03:39 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161819, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:03:39.570468 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.549791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.28.251.254:43523)
0808 04:03:39.550415 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:39.550421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:39.551499 (+ 1078us) server_negotiation.cc:407] Connection header received
0808 04:03:39.554117 (+ 2618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:39.554122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:39.554200 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:39.554303 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:39.556319 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.557445 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.559157 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.559764 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.559906 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:39.563879 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:39.563885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:39.563889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:39.563949 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:39.566830 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:39.567853 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:39.567858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:39.567860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:39.567998 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:39.568539 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:39.568543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:39.568546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:39.568856 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:39.568972 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:39.569961 (+ 989us) server_negotiation.cc:299] Negotiation successful
0808 04:03:39.570156 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":283,"threads_started":1}
I20260808 04:03:39.572073 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.550386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43523 (local address 127.0.0.1:46100)
0808 04:03:39.551264 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.551330 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:39.553753 (+ 2423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:39.554607 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:39.554620 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:39.555354 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:39.556045 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:39.556065 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.557649 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.557654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:39.558978 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:39.558992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.560010 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.560015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:39.560249 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:39.560949 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:39.560976 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:39.563544 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:39.567010 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:39.567022 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:39.567042 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:39.567712 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:39.568123 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:39.568129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:39.568133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:39.568426 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:39.568984 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:39.568993 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:39.569463 (+ 470us) client_negotiation.cc:769] Sending connection context
0808 04:03:39.569877 (+ 414us) client_negotiation.cc:240] Negotiation successful
0808 04:03:39.570494 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":317,"threads_started":1}
I20260808 04:03:39.574179 3227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.251.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.251.252@KRBTEST.COM" created.
Aug 08 04:03:39 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161819, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
I20260808 04:03:39.590162 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.575106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40310)
0808 04:03:39.575314 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.575340 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:39.575497 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:39.576375 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:39.576379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:39.576779 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:39.576961 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:39.576971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.578431 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.578437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:39.579732 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:39.579751 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.580712 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.580719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:39.580942 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:39.581629 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:39.581657 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:39.584291 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:39.587629 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:39.587635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:39.587640 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:39.588015 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:39.588409 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:39.588414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:39.588418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:39.588544 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:39.589181 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:39.589186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:39.589342 (+ 156us) client_negotiation.cc:769] Sending connection context
0808 04:03:39.589457 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 04:03:39.589750 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 04:03:39.590233 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.575123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.28.251.253:41283)
0808 04:03:39.575819 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:39.575838 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:39.575876 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:03:39.575939 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:39.575947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:39.576211 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:39.576345 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:39.577121 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.578260 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.579894 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.580535 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.580635 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:39.584546 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:39.584561 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:39.584576 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:39.584616 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:39.587441 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:39.588156 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:39.588163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:39.588166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:39.588292 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:39.588654 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:39.588658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:39.588660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:39.589033 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:39.589250 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:39.589640 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 04:03:39.589871 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":323,"threads_started":1}
I20260808 04:03:39.593140 3227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:03:39.594676 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.594042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48834)
0808 04:03:39.594248 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.594437 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260808 04:03:39.595012 3155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
I20260808 04:03:39.596904 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.596491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48846)
0808 04:03:39.596674 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.596776 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260808 04:03:39.597144 3227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:41245: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.251.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab.
Entry for principal kudu/127.28.251.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab.
I20260808 04:03:39.622884 29679 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhSn85p/build/asan/bin/kudu
/tmp/dist-test-taskhSn85p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.252:41245
--webserver_interface=127.28.251.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab
--principal=kudu/127.28.251.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.251.212:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.251.254:43523,127.28.251.253:41283,127.28.251.252:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:03:39.646520 3227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } attempt: 1
I20260808 04:03:39.647816 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.647284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48862)
0808 04:03:39.647521 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.647638 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 04:03:39.649045 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.648635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48872)
0808 04:03:39.648817 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.648925 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260808 04:03:39.649273 3227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:41245: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
I20260808 04:03:39.659605 3114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } attempt: 5
Aug 08 04:03:39 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161818, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
I20260808 04:03:39.673182 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.660353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40324)
0808 04:03:39.660551 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.660574 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:39.660710 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:39.661082 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:39.661087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:39.661503 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:39.661707 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:39.661717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.662739 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.662745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:39.663689 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:39.663704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.664467 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.664472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:39.664688 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:39.665322 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:39.665340 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:39.667595 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:39.670628 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:39.670634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:39.670638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:39.671016 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:39.671405 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:39.671411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:39.671415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:39.671602 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:39.672144 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:39.672149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:39.672313 (+ 164us) client_negotiation.cc:769] Sending connection context
0808 04:03:39.672438 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 04:03:39.672771 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 04:03:39.673167 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.660467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.28.251.253:41283)
0808 04:03:39.660710 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:39.660716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:39.660742 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:03:39.660834 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:39.660839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:39.660930 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:39.661020 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:39.661863 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.662598 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.663841 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:39.664307 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:39.664426 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:39.667796 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:39.667802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:39.667806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:39.667844 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:39.670470 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:39.671140 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:39.671156 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:39.671159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:39.671290 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:39.671715 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:39.671719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:39.671721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:39.672021 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:39.672163 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:39.672590 (+ 427us) server_negotiation.cc:299] Negotiation successful
0808 04:03:39.672831 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 04:03:39.674634 3114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:03:39.675858 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.675443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48888)
0808 04:03:39.675637 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.675719 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 04:03:39.676869 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.676548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48890)
0808 04:03:39.676672 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.676761 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 04:03:39.677059 3114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:41245: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
I20260808 04:03:39.717515 3227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } attempt: 2
I20260808 04:03:39.718806 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.718288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48904)
0808 04:03:39.718507 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.718608 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 04:03:39.720001 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.719586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48906)
0808 04:03:39.719782 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.719867 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260808 04:03:39.720286 3227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:41245: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
I20260808 04:03:39.728271 3114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } attempt: 1
I20260808 04:03:39.729480 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.728995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48908)
0808 04:03:39.729198 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.729296 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 04:03:39.730484 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.730130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48916)
0808 04:03:39.730278 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.730369 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:03:39.730702 3114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:41245: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
I20260808 04:03:39.798936 3114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } attempt: 2
I20260808 04:03:39.800184 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.799667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48918)
0808 04:03:39.799901 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.799998 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 04:03:39.801169 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.800836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48924)
0808 04:03:39.800986 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.801055 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 04:03:39.801378 3114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:41245: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
I20260808 04:03:39.811564 3227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } attempt: 3
I20260808 04:03:39.812860 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.812319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48934)
0808 04:03:39.812540 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.812655 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 04:03:39.813935 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.813484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48950)
0808 04:03:39.813735 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.813816 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260808 04:03:39.814153 3227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:41245: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
I20260808 04:03:39.907617 3114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } attempt: 3
I20260808 04:03:39.909003 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.908414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48966)
0808 04:03:39.908692 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.908809 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260808 04:03:39.910138 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.909696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48970)
0808 04:03:39.909920 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.910009 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260808 04:03:39.910386 3114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:41245: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
I20260808 04:03:39.957389 3227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } attempt: 4
I20260808 04:03:39.958753 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.958130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48978)
0808 04:03:39.958439 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.958556 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 04:03:39.959991 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:39.959658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48980)
0808 04:03:39.959802 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:39.959879 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 04:03:39.960227 3227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:41245: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
W20260808 04:03:39.997104 3236 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:03:39.997514 3236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:03:39.997619 3236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:03:39.997740 3236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:03:40.005415 3236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:03:40.005544 3236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:03:40.005636 3236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:03:40.005704 3236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:03:40.005808 3236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:03:40.016749 3236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.251.212:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.251.254:43523,127.28.251.253:41283,127.28.251.252:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/krb5kdc/kudu_127.28.251.252.keytab
--principal=kudu/127.28.251.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.251.252:41245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.251.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 02:38:26 UTC on ccycloud.quasar-koaxys.root.comops.site
build id 4574019
ASAN enabled
I20260808 04:03:40.018193 3236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:03:40.019672 3236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:03:40.032294 3249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:40.032327 3243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:03:40.032187 3246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:03:40.033634 3236 server_base.cc:1034] running on GCE node
I20260808 04:03:40.034590 3236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:03:40.036316 3236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:03:40.037628 3236 hybrid_clock.cc:648] HybridClock initialized: now 1786161820037528 us; error 53 us; skew 500 ppm
I20260808 04:03:40.038066 3236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 04:03:40 dist-test-slave-pww0 krb5kdc[2992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161820, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:03:40.041961 3236 init.cc:377] Logged in from keytab as kudu/127.28.251.252@KRBTEST.COM (short username kudu)
I20260808 04:03:40.043908 3236 webserver.cc:460] Webserver started at http://127.28.251.252:44929/ using document root <none> and password file <none>
I20260808 04:03:40.044736 3236 fs_manager.cc:359] Metadata directory not provided
I20260808 04:03:40.044859 3236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:03:40.045198 3236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:03:40.048936 3236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/data/instance:
uuid: "1163d4ffc2ec4dd18dd88705007cc3ca"
format_stamp: "Formatted at 2026-08-08 04:03:40 on dist-test-slave-pww0"
server_key: "0fee0215d5e582d5e32edc349b1caa1d"
server_key_iv: "a9c29d0eb0aa356cc3d8e287e7d74e41"
server_key_version: "encryptionkey@0"
I20260808 04:03:40.049832 3236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/wal/instance:
uuid: "1163d4ffc2ec4dd18dd88705007cc3ca"
format_stamp: "Formatted at 2026-08-08 04:03:40 on dist-test-slave-pww0"
server_key: "0fee0215d5e582d5e32edc349b1caa1d"
server_key_iv: "a9c29d0eb0aa356cc3d8e287e7d74e41"
server_key_version: "encryptionkey@0"
I20260808 04:03:40.056694 3236 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260808 04:03:40.060652 3261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:03:40.062286 3236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:03:40.062492 3236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/data,/tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/wal
uuid: "1163d4ffc2ec4dd18dd88705007cc3ca"
format_stamp: "Formatted at 2026-08-08 04:03:40 on dist-test-slave-pww0"
server_key: "0fee0215d5e582d5e32edc349b1caa1d"
server_key_iv: "a9c29d0eb0aa356cc3d8e287e7d74e41"
server_key_version: "encryptionkey@0"
I20260808 04:03:40.062739 3236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:03:40.084146 3236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:03:40.087565 3114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } attempt: 4
I20260808 04:03:40.088621 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.088168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:48988)
0808 04:03:40.088374 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:40.088465 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 04:03:40.088851 3236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:03:40.089118 3236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:03:40.089627 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.089275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:49000)
0808 04:03:40.089423 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:40.089495 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 04:03:40.089840 3114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.251.252:41245: Network error: Client connection negotiation failed: client connection to 127.28.251.252:41245: connect: Connection refused (error 111)
I20260808 04:03:40.108809 3236 rpc_server.cc:307] RPC server started. Bound to: 127.28.251.252:41245
I20260808 04:03:40.108816 3339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.251.252:41245 every 8 connection(s)
I20260808 04:03:40.110270 3236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/data/info.pb
I20260808 04:03:40.112789 29679 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhSn85p/build/asan/bin/kudu as pid 3236
I20260808 04:03:40.113036 29679 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhSn85p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786161430375636-29679-0/minicluster-data/master-2/wal/instance
I20260808 04:03:40.113673 29679 external_mini_cluster.cc:1428] Setting key 25c4283fffcfa8ffc904f61eb1368037
I20260808 04:03:40.115052 3340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:03:40.123605 3340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:03:40 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
Aug 08 04:03:40 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161820, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
I20260808 04:03:40.136881 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.115705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.28.251.252:41245)
0808 04:03:40.116528 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.116540 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.116583 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:03:40.118204 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.118227 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.118660 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.119050 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.120622 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.122023 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.123415 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.124239 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.124488 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.129109 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:40.129121 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:40.129135 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:40.129180 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:40.132254 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.133076 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.133084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.133090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.133284 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.133687 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.133691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.133693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.134382 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:40.134729 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:40.135191 (+ 462us) server_negotiation.cc:299] Negotiation successful
0808 04:03:40.135692 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":354,"threads_started":1}
I20260808 04:03:40.141646 3348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.126399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.28.251.254:43523)
0808 04:03:40.126957 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.126964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.127497 (+ 533us) server_negotiation.cc:407] Connection header received
0808 04:03:40.127730 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.127736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.127852 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.127975 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.129342 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.130059 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.131645 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.132072 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.132190 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.135748 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:40.135754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:40.135757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:40.135794 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:40.138500 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.139281 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.139286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.139288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.139494 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.139925 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.139929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.139931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.140299 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:40.140485 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:40.141107 (+ 622us) server_negotiation.cc:299] Negotiation successful
0808 04:03:40.141352 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":244,"threads_started":1}
I20260808 04:03:40.141708 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.126327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.254:43523 (local address 127.0.0.1:46114)
0808 04:03:40.127322 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:40.127368 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:40.127586 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:40.128121 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:40.128129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:40.128807 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:40.129163 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:40.129175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.130232 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.130235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:40.131444 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:40.131497 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.132272 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.132278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:40.132448 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.133047 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:40.133065 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:40.135550 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:40.138645 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:40.138658 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:40.138666 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:40.139146 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:40.139630 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:40.139634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:40.139636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:40.139797 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:40.140572 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:40.140580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:40.140770 (+ 190us) client_negotiation.cc:769] Sending connection context
0808 04:03:40.140929 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 04:03:40.141271 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":561,"threads_started":1}
I20260808 04:03:40.143226 3340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:03:40 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.254@KRBTEST.COM
Aug 08 04:03:40 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161820, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
I20260808 04:03:40.156744 3351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.140166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.28.251.254:43523)
0808 04:03:40.140768 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.140781 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.140800 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:03:40.140840 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.140844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.140897 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.141007 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.142801 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.143707 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.145433 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.146042 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.146171 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.150965 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:40.150971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:40.150974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:40.151011 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:40.153741 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.154448 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.154454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.154458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.154629 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.155073 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.155078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.155081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.155439 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:40.155694 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:40.156099 (+ 405us) server_negotiation.cc:299] Negotiation successful
0808 04:03:40.156370 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":276,"threads_started":1}
I20260808 04:03:40.158141 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.144082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.253:41283 (local address 127.0.0.1:40326)
0808 04:03:40.144429 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:40.144447 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:40.144567 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:40.144862 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:40.144867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:40.145345 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:40.145523 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:40.145532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.146736 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.146741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:40.148092 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:40.148111 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.148903 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.148910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:40.149124 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.149720 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:40.149738 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:40.151922 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:40.155303 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:40.155307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:40.155309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:40.155893 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:40.156351 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:40.156356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:40.156360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:40.156545 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:40.157150 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:40.157155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:40.157331 (+ 176us) client_negotiation.cc:769] Sending connection context
0808 04:03:40.157451 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 04:03:40.157762 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260808 04:03:40.158171 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.144088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.28.251.253:41283)
0808 04:03:40.144275 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.144281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.144528 (+ 247us) server_negotiation.cc:407] Connection header received
0808 04:03:40.144669 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.144673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.144739 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.144823 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.145670 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.146582 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.148282 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.148738 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.148847 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.152144 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:40.152150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:40.152153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:40.152190 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:40.155140 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.156031 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.156036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.156039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.156211 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.156680 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.156684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.156686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.157008 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:40.157175 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:40.157587 (+ 412us) server_negotiation.cc:299] Negotiation successful
0808 04:03:40.157809 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:03:40.159170 3340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:03:40 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161820, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.252@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
Aug 08 04:03:40 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.251.253@KRBTEST.COM
I20260808 04:03:40.173382 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.159995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:49032)
0808 04:03:40.160153 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:40.160170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:40.160262 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:40.160665 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:40.160668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:40.161169 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:40.161378 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:40.161391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.162586 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.162590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:40.163723 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:40.163739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.164920 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.164924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:40.165070 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.165635 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:40.165651 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:40.168009 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:40.170986 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:40.170991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:40.170996 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:40.171367 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:40.171756 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:40.171761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:40.171765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:40.171893 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:40.172380 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:40.172384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:40.172532 (+ 148us) client_negotiation.cc:769] Sending connection context
0808 04:03:40.172652 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 04:03:40.172973 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 04:03:40.173533 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.160210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.28.251.252:41245)
0808 04:03:40.160409 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.160414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.160433 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:03:40.160480 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.160484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.160554 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.160655 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.161517 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.162456 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.163877 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.164526 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.164659 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.168209 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:40.168215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:40.168218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:40.168251 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:40.170852 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.171502 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.171507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.171510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.171635 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.171992 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.171995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.171997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.172268 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:40.172400 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:40.172758 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 04:03:40.173158 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":74}
I20260808 04:03:40.175031 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.158715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.28.251.253:41283)
0808 04:03:40.158886 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.158891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.159007 (+ 116us) server_negotiation.cc:407] Connection header received
0808 04:03:40.159185 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.159189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.159249 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.159338 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.160759 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.161752 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.163199 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.163861 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.163966 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.168943 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:40.168949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:40.168952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:40.168989 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:40.172009 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.172828 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.172834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.172837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.173059 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.173516 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.173521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.173523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.173903 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:40.174224 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:40.174467 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 04:03:40.174717 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:03:40.180253 3340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca: Bootstrap starting.
I20260808 04:03:40.180765 29679 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:03:40.184090 3340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:40.185343 3340 log.cc:824] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:40.183377 3074 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46116
I20260808 04:03:40.188664 3340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca: No bootstrap required, opened a new log
I20260808 04:03:40.199872 3340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } }
I20260808 04:03:40.200304 3340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1163d4ffc2ec4dd18dd88705007cc3ca, State: Initialized, Role: FOLLOWER
I20260808 04:03:40.200893 3340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } }
I20260808 04:03:40.202029 3355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } } }
I20260808 04:03:40.202379 3355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:40.203313 3340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:03:40.213958 3376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:03:40.214128 3376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:03:40.211421 3236 master_runner.cc:186] Error getting master registration for 127.28.251.254:43523: OK, instance_id { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" instance_seqno: 1786161818946495 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:03:40.223524 3236 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:03:40.259500 3227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } attempt: 5
Aug 08 04:03:40 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161819, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.253@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
I20260808 04:03:40.277546 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.260329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:49042)
0808 04:03:40.260570 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:40.260593 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:40.260733 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:40.261104 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:40.261110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:40.261667 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:40.261888 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:40.261900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.263405 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.263410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:40.264918 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:40.264934 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.265964 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.265970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:40.266205 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.266937 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:40.266962 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:40.269995 (+ 3033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:40.274407 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:40.274414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:40.274418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:40.275040 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:40.275531 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:40.275537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:40.275541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:40.275752 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:40.276462 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:40.276467 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:40.276644 (+ 177us) client_negotiation.cc:769] Sending connection context
0808 04:03:40.276809 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 04:03:40.277143 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 04:03:40.277563 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.260423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.28.251.252:41245)
0808 04:03:40.260724 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.260731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.260756 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:03:40.260863 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.260869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.260960 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.261077 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.262079 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.263269 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.265090 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.265798 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.265934 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.270259 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:40.270268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:40.270273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:40.270322 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:40.274211 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.275192 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.275198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.275202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.275374 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.275876 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.275882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.275885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.276318 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:40.276516 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:40.276931 (+ 415us) server_negotiation.cc:299] Negotiation successful
0808 04:03:40.277172 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260808 04:03:40.282507 3227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202: Bootstrap starting.
I20260808 04:03:40.286784 3227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:40.288019 3227 log.cc:824] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:40.290961 3227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202: No bootstrap required, opened a new log
I20260808 04:03:40.302403 3227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } }
I20260808 04:03:40.302874 3227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b136c297e94274a4177478ff74e202, State: Initialized, Role: FOLLOWER
I20260808 04:03:40.303409 3227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } }
I20260808 04:03:40.304673 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } } }
I20260808 04:03:40.305197 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:40.305958 3227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:03:40.323060 3404 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:03:40.323240 3404 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:03:40.319721 3122 master_runner.cc:186] Error getting master registration for 127.28.251.254:43523: OK, instance_id { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" instance_seqno: 1786161818946495 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:03:40.331946 3122 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 04:03:40.341473 3269 tablet.cc:2369] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:03:40.367056 3114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } attempt: 5
Aug 08 04:03:40 dist-test-slave-pww0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786161818, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.251.254@KRBTEST.COM for kudu/127.28.251.252@KRBTEST.COM
I20260808 04:03:40.386159 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.367881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.251.252:41245 (local address 127.0.0.1:49046)
0808 04:03:40.368126 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 04:03:40.368151 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 04:03:40.368301 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:03:40.368909 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:03:40.368915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:03:40.369449 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:03:40.369837 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:40.369856 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.371429 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.371436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:40.372973 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:03:40.372992 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.374084 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.374090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:03:40.374299 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.375036 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:03:40.375061 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:03:40.378111 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:03:40.382910 (+ 4799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:40.382917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:40.382922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:40.383558 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:40.384101 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:03:40.384106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:03:40.384110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:03:40.384317 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:03:40.385049 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:03:40.385054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:03:40.385250 (+ 196us) client_negotiation.cc:769] Sending connection context
0808 04:03:40.385404 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 04:03:40.385721 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 04:03:40.386219 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.368355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.28.251.252:41245)
0808 04:03:40.368534 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.368540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.368562 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:03:40.368631 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.368637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.368743 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.368854 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.370042 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.371257 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.373184 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.373911 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.374052 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.378395 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:40.378403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:40.378409 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:40.378461 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:40.382688 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.383748 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.383754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.383758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.383945 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.384465 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.384471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.384475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.384905 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:40.385083 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:40.385545 (+ 462us) server_negotiation.cc:299] Negotiation successful
0808 04:03:40.385828 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:03:40.393090 3114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747: Bootstrap starting.
I20260808 04:03:40.397964 3114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747: Neither blocks nor log segments found. Creating new log.
I20260808 04:03:40.399669 3114 log.cc:824] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747: Log is configured to *not* fsync() on all Append() calls
I20260808 04:03:40.404280 3114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747: No bootstrap required, opened a new log
I20260808 04:03:40.412773 3114 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } }
I20260808 04:03:40.413316 3114 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad97d3d9cd454ddd9b2dc0b5d3e22747, State: Initialized, Role: FOLLOWER
I20260808 04:03:40.413926 3114 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } }
I20260808 04:03:40.415364 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } } }
I20260808 04:03:40.416380 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:40.416945 3114 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:03:40.417044 3074 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 04:03:40.417261 3074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:40.421008 3074 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } }
I20260808 04:03:40.423889 3194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "22b136c297e94274a4177478ff74e202"
I20260808 04:03:40.424515 3194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:03:40.429015 3194 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad97d3d9cd454ddd9b2dc0b5d3e22747 in term 1.
I20260808 04:03:40.429993 3042 leader_election.cc:302] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b136c297e94274a4177478ff74e202, ad97d3d9cd454ddd9b2dc0b5d3e22747; no voters:
I20260808 04:03:40.431959 3303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca"
I20260808 04:03:40.432542 3303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [term 0 FOLLOWER]: Advancing to term 1
W20260808 04:03:40.436429 3049 tablet.cc:2369] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:03:40.437139 3303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad97d3d9cd454ddd9b2dc0b5d3e22747 in term 1.
I20260808 04:03:40.438086 3074 leader_election.cc:288] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [CANDIDATE]: Term 1 election: Requested vote from peers 22b136c297e94274a4177478ff74e202 (127.28.251.253:41283), 1163d4ffc2ec4dd18dd88705007cc3ca (127.28.251.252:41245)
I20260808 04:03:40.445284 3408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:03:40.455353 3408 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [term 1 LEADER]: Becoming Leader. State: Replica: ad97d3d9cd454ddd9b2dc0b5d3e22747, State: Running, Role: LEADER
I20260808 04:03:40.456594 3408 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } }
I20260808 04:03:40.473032 3418 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad97d3d9cd454ddd9b2dc0b5d3e22747. Latest consensus state: current_term: 1 leader_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } } }
I20260808 04:03:40.473703 3418 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:40.484091 3430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:03:40.489755 3432 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:03:40.507133 3430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260808 04:03:40.514590 3160 tablet.cc:2369] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:03:40.537920 3351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.488746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.28.251.254:43523)
0808 04:03:40.488957 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.488965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.492856 (+ 3891us) server_negotiation.cc:407] Connection header received
0808 04:03:40.492955 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.492960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.493055 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.493165 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.511909 (+ 18744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.513021 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.519589 (+ 6568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.520309 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.520422 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.524273 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:40.524279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:40.524284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:40.524333 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:40.528315 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.529309 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.529315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.529319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.529499 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.535662 (+ 6163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.535670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.535673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.536195 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:40.536485 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:40.537242 (+ 757us) server_negotiation.cc:299] Negotiation successful
0808 04:03:40.537535 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:03:40.559041 3303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [term 1 FOLLOWER]: Refusing update from remote peer ad97d3d9cd454ddd9b2dc0b5d3e22747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:03:40.560621 3418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:03:40.568943 3194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [term 1 FOLLOWER]: Refusing update from remote peer ad97d3d9cd454ddd9b2dc0b5d3e22747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:03:40.570501 3418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:03:40.593706 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.492974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.28.251.252:41245)
0808 04:03:40.493732 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.493739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.512511 (+ 18772us) server_negotiation.cc:407] Connection header received
0808 04:03:40.513264 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.513269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.513391 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.513472 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.519471 (+ 5999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.520561 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.527444 (+ 6883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.528302 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.528430 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.584553 (+ 56123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:40.584562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:40.584568 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:40.584626 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:40.589083 (+ 4457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.590542 (+ 1459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.590549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.590557 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.590737 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.591270 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.591276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.591280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.591761 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:40.592036 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:40.592964 (+ 928us) server_negotiation.cc:299] Negotiation successful
0808 04:03:40.593249 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562}
I20260808 04:03:40.603549 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:40.509307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.28.251.253:41283)
0808 04:03:40.509552 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.509560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.516504 (+ 6944us) server_negotiation.cc:407] Connection header received
0808 04:03:40.516995 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.517001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.517091 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.517182 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.524256 (+ 7074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.525387 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.539568 (+ 14181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:40.541403 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:40.541550 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:40.594941 (+ 53391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:03:40.594949 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:03:40.594954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:03:40.595003 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:03:40.599071 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.600596 (+ 1525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.600603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.600606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.600789 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:03:40.601296 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:03:40.601302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:03:40.601306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:03:40.601753 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:03:40.602016 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:40.602690 (+ 674us) server_negotiation.cc:299] Negotiation successful
0808 04:03:40.602986 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260808 04:03:40.623545 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad97d3d9cd454ddd9b2dc0b5d3e22747. Latest consensus state: current_term: 1 leader_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } } }
I20260808 04:03:40.623944 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:40.629525 3440 mvcc.cc:205] Tried to move back new op lower bound from 7316118816937148416 to 7316118816614010880. Current Snapshot: MvccSnapshot[applied={T|T < 7316118816937148416}]
I20260808 04:03:40.636569 3445 mvcc.cc:205] Tried to move back new op lower bound from 7316118816937148416 to 7316118816614010880. Current Snapshot: MvccSnapshot[applied={T|T < 7316118816937148416}]
I20260808 04:03:40.646634 3355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad97d3d9cd454ddd9b2dc0b5d3e22747. Latest consensus state: current_term: 1 leader_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } } }
I20260808 04:03:40.647008 3355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:40.650722 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } } }
I20260808 04:03:40.651033 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:40.664212 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } } }
I20260808 04:03:40.664572 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:40.664892 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } } }
I20260808 04:03:40.665163 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747 [sys.catalog]: This master's current role is: LEADER
I20260808 04:03:40.668236 3355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad97d3d9cd454ddd9b2dc0b5d3e22747" member_type: VOTER last_known_addr { host: "127.28.251.254" port: 43523 } } peers { permanent_uuid: "22b136c297e94274a4177478ff74e202" member_type: VOTER last_known_addr { host: "127.28.251.253" port: 41283 } } peers { permanent_uuid: "1163d4ffc2ec4dd18dd88705007cc3ca" member_type: VOTER last_known_addr { host: "127.28.251.252" port: 41245 } } }
I20260808 04:03:40.668531 3355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:03:40.675828 3430 catalog_manager.cc:1347] Generated new cluster ID: d5c812f633e147d89c723369e51d8434
I20260808 04:03:40.675957 3430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:03:40.700953 3430 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:03:40.703037 3430 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:03:40.725937 3430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad97d3d9cd454ddd9b2dc0b5d3e22747: Generated new TSK 0
I20260808 04:03:40.727161 3430 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:03:40.840883 3351 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:40.825480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.28.251.254:43523)
0808 04:03:40.825900 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.825906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.835567 (+ 9661us) server_negotiation.cc:407] Connection header received
0808 04:03:40.836842 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.836850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.836950 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.837030 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.840565 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46136: BlockingRecv error: recv got EOF from 127.0.0.1:46136 (error 108)
Metrics: {"server-negotiator.queue_time_us":300}
W20260808 04:03:40.842242 3347 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:40.832069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.28.251.252:41245)
0808 04:03:40.832282 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.832289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.840301 (+ 8012us) server_negotiation.cc:407] Connection header received
0808 04:03:40.840592 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.840599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.840713 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.840798 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.841983 (+ 1185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49062: BlockingRecv error: recv got EOF from 127.0.0.1:49062 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260808 04:03:40.843971 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:40.825698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.28.251.253:41283)
0808 04:03:40.825893 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.825899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.839403 (+ 13504us) server_negotiation.cc:407] Connection header received
0808 04:03:40.841554 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.841560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.841677 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.841757 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.843726 (+ 1969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40358: BlockingRecv error: recv got EOF from 127.0.0.1:40358 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 04:03:40.860469 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:40.857862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.28.251.253:41283)
0808 04:03:40.858074 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.858081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.858630 (+ 549us) server_negotiation.cc:407] Connection header received
0808 04:03:40.858895 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.858902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.859064 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.859190 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.860124 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40360: BlockingRecv error: recv got EOF from 127.0.0.1:40360 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260808 04:03:40.867607 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:40.865075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.28.251.253:41283)
0808 04:03:40.865270 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.865276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.865788 (+ 512us) server_negotiation.cc:407] Connection header received
0808 04:03:40.866016 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.866022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.866119 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.866234 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.867286 (+ 1052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40362: BlockingRecv error: recv got EOF from 127.0.0.1:40362 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 04:03:40.875803 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:40.873320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.28.251.253:41283)
0808 04:03:40.873517 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.873527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.874088 (+ 561us) server_negotiation.cc:407] Connection header received
0808 04:03:40.874316 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.874321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.874455 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.874585 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.875513 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40370: BlockingRecv error: recv got EOF from 127.0.0.1:40370 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 04:03:40.885762 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:40.883353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.28.251.253:41283)
0808 04:03:40.883563 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.883570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.884117 (+ 547us) server_negotiation.cc:407] Connection header received
0808 04:03:40.884328 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.884334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.884446 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.884572 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.885475 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40382: BlockingRecv error: recv got EOF from 127.0.0.1:40382 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 04:03:40.900420 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:40.897785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40394 (local address 127.28.251.253:41283)
0808 04:03:40.898015 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.898022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.898630 (+ 608us) server_negotiation.cc:407] Connection header received
0808 04:03:40.898897 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.898903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.899017 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.899153 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.900141 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40394: BlockingRecv error: recv got EOF from 127.0.0.1:40394 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260808 04:03:40.922864 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:40.920436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.28.251.253:41283)
0808 04:03:40.920637 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.920643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.921243 (+ 600us) server_negotiation.cc:407] Connection header received
0808 04:03:40.921443 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.921450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.921565 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.921700 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.922588 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40408: BlockingRecv error: recv got EOF from 127.0.0.1:40408 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 04:03:40.975173 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:40.972668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.28.251.253:41283)
0808 04:03:40.972899 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:40.972906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:40.973499 (+ 593us) server_negotiation.cc:407] Connection header received
0808 04:03:40.973700 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:40.973706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:40.973832 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:40.973957 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:40.974881 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40410: BlockingRecv error: recv got EOF from 127.0.0.1:40410 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 04:03:41.044865 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:41.042933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40424 (local address 127.28.251.253:41283)
0808 04:03:41.043228 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:41.043235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:41.043496 (+ 261us) server_negotiation.cc:407] Connection header received
0808 04:03:41.043661 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:41.043666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:41.043776 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:41.043889 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:41.044609 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40424: BlockingRecv error: recv got EOF from 127.0.0.1:40424 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260808 04:03:41.179201 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:03:41.177184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.28.251.253:41283)
0808 04:03:41.177467 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:41.177473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:41.177728 (+ 255us) server_negotiation.cc:407] Connection header received
0808 04:03:41.177977 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:41.177982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:41.178063 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:41.178154 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:03:41.178992 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40436: BlockingRecv error: recv got EOF from 127.0.0.1:40436 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260808 04:03:41.216464 3376 catalog_manager.cc:1259] Loaded cluster ID: d5c812f633e147d89c723369e51d8434
I20260808 04:03:41.216572 3376 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca: loading cluster ID for follower catalog manager: success
I20260808 04:03:41.219488 3376 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca: acquiring CA information for follower catalog manager: success
I20260808 04:03:41.221472 3376 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1163d4ffc2ec4dd18dd88705007cc3ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:03:41.325445 3404 catalog_manager.cc:1259] Loaded cluster ID: d5c812f633e147d89c723369e51d8434
I20260808 04:03:41.325560 3404 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202: loading cluster ID for follower catalog manager: success
I20260808 04:03:41.328250 3404 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202: acquiring CA information for follower catalog manager: success
I20260808 04:03:41.330229 3404 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 22b136c297e94274a4177478ff74e202: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:03:41.446075 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:41.440140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.28.251.253:41283)
0808 04:03:41.440384 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:41.440390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:41.440659 (+ 269us) server_negotiation.cc:407] Connection header received
0808 04:03:41.440895 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:41.440904 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:41.441078 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:41.441187 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:41.441658 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:41.442409 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:41.443906 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:41.444325 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:41.444445 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:41.444893 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:41.445451 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:41.445564 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:41.445665 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 04:03:41.445763 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:03:41.458993 3554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:41.452312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.28.251.252:41245)
0808 04:03:41.453312 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:41.453317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:41.453335 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:03:41.453437 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:41.453441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:41.453617 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:41.453786 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:41.454247 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:41.455040 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:41.456523 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:41.456968 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:41.457077 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:41.457652 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:41.458261 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:41.458377 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:41.458547 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 04:03:41.458640 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":766,"threads_started":1}
I20260808 04:03:41.485100 3554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:41.466430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.28.251.252:41245)
0808 04:03:41.472258 (+ 5828us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:41.472268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:41.478933 (+ 6665us) server_negotiation.cc:407] Connection header received
0808 04:03:41.479026 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:41.479032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:41.479128 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:41.479229 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:41.480037 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:41.481032 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:41.482860 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:41.483577 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:41.483696 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:41.484291 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:41.484443 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:41.484541 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:41.484701 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 04:03:41.484797 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5600}
I20260808 04:03:41.485564 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:41.466267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.28.251.253:41283)
0808 04:03:41.466549 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:41.466558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:41.479199 (+ 12641us) server_negotiation.cc:407] Connection header received
0808 04:03:41.479385 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:41.479389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:41.479502 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:41.479594 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:41.480272 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:41.481234 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:41.483407 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:41.484009 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:41.484171 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:41.484695 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:41.484814 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:41.484929 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:41.485146 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 04:03:41.485350 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260808 04:03:41.487028 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:03:41.465885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.28.251.254:43523)
0808 04:03:41.477323 (+ 11438us) server_negotiation.cc:206] Beginning negotiation
0808 04:03:41.477331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:03:41.477889 (+ 558us) server_negotiation.cc:407] Connection header received
0808 04:03:41.478176 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:03:41.478184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:03:41.478458 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:03:41.479394 (+ 936us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:03:41.479599 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:41.480665 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:41.482919 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:03:41.483559 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:03:41.483696 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:03:41.484501 (+ 805us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:03:41.485219 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:03:41.486479 (+ 1260us) server_negotiation.cc:1035] Waiting for connection context
0808 04:03:41.486615 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 04:03:41.486719 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11268,"thread_start_us":11113,"threads_started":1}
I20260808 04:03:41.491930 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 3017
I20260808 04:03:41.493149 3108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:41.606348 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 3017
I20260808 04:03:41.640903 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 3122
I20260808 04:03:41.642304 3221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:41.755314 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 3122
I20260808 04:03:41.786350 29679 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhSn85p/build/asan/bin/kudu pid 3236
I20260808 04:03:41.787635 3332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 04:03:41.897953 29679 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhSn85p/build/asan/bin/kudu with pid 3236
2026-08-08T04:03:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5123 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10176 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (391494 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST